[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 00:59:32 dist-test-slave-7v10 krb5kdc[3379](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 00:59:32 dist-test-slave-7v10 krb5kdc[3379](info): set up 2 sockets
Sep 26 00:59:32 dist-test-slave-7v10 krb5kdc[3379](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260926 00:59:33.106163 3372 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Sep 26 00:59:33 dist-test-slave-7v10 krb5kdc[3379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T00:59:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T00:59:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.75.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab.
Entry for principal kudu/127.3.75.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab.
I20260926 00:59:33.221565 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:34963
--webserver_interface=127.3.75.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:42701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.75.62:34963
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 00:59:33.303194 3392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 00:59:33.303385 3392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 00:59:33.303406 3392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 00:59:33.304558 3392 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 00:59:33.304600 3392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 00:59:33.304615 3392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 00:59:33.304631 3392 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 00:59:33.304643 3392 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 00:59:33.306222 3392 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:42701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.75.62:34963
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:34963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 00:59:33.306461 3392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 00:59:33.306691 3392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 00:59:33.311764 3397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:33.311772 3400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:33.311781 3398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 00:59:33.311858 3392 server_base.cc:1034] running on GCE node
I20260926 00:59:33.312153 3392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 00:59:33.312381 3392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 00:59:33.313552 3392 hybrid_clock.cc:648] HybridClock initialized: now 1790384373313535 us; error 28 us; skew 500 ppm
Sep 26 00:59:33 dist-test-slave-7v10 krb5kdc[3379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384373, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 00:59:33.316047 3392 init.cc:377] Logged in from keytab as kudu/127.3.75.62@KRBTEST.COM (short username kudu)
I20260926 00:59:33.316457 3392 webserver.cc:460] Webserver started at http://127.3.75.62:35019/ using document root <none> and password file <none>
I20260926 00:59:33.316705 3392 fs_manager.cc:359] Metadata directory not provided
I20260926 00:59:33.316773 3392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 00:59:33.316911 3392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 00:59:33.318027 3392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/master-0/data/instance:
uuid: "79d75f67ca8c4b3cbdd88c4743f7b09e"
format_stamp: "Formatted at 2026-09-26 00:59:33 on dist-test-slave-7v10"
server_key: "082b27598969603614ae68d9e09d667f"
server_key_iv: "18ad3b7d69b9dca36d6f0788a7bc9204"
server_key_version: "encryptionkey@0"
I20260926 00:59:33.318619 3392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/master-0/wal/instance:
uuid: "79d75f67ca8c4b3cbdd88c4743f7b09e"
format_stamp: "Formatted at 2026-09-26 00:59:33 on dist-test-slave-7v10"
server_key: "082b27598969603614ae68d9e09d667f"
server_key_iv: "18ad3b7d69b9dca36d6f0788a7bc9204"
server_key_version: "encryptionkey@0"
I20260926 00:59:33.320221 3392 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 00:59:33.321010 3407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:33.321300 3392 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 00:59:33.321408 3392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/master-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/master-0/wal
uuid: "79d75f67ca8c4b3cbdd88c4743f7b09e"
format_stamp: "Formatted at 2026-09-26 00:59:33 on dist-test-slave-7v10"
server_key: "082b27598969603614ae68d9e09d667f"
server_key_iv: "18ad3b7d69b9dca36d6f0788a7bc9204"
server_key_version: "encryptionkey@0"
I20260926 00:59:33.321522 3392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 00:59:33.335251 3392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 00:59:33.337447 3392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 00:59:33.337613 3392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 00:59:33.341907 3392 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.62:34963
I20260926 00:59:33.341934 3459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.62:34963 every 8 connection(s)
I20260926 00:59:33.342295 3392 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
I20260926 00:59:33.343201 3460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 00:59:33.345399 3460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79d75f67ca8c4b3cbdd88c4743f7b09e: Bootstrap starting.
I20260926 00:59:33.346048 3460 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79d75f67ca8c4b3cbdd88c4743f7b09e: Neither blocks nor log segments found. Creating new log.
I20260926 00:59:33.346140 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 3392
I20260926 00:59:33.346261 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/master-0/wal/instance
I20260926 00:59:33.346401 3460 log.cc:824] T 00000000000000000000000000000000 P 79d75f67ca8c4b3cbdd88c4743f7b09e: Log is configured to *not* fsync() on all Append() calls
I20260926 00:59:33.346990 3372 external_mini_cluster.cc:1428] Setting key 22010d73a3434a1c3e8442f3cab74c55
I20260926 00:59:33.347162 3460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79d75f67ca8c4b3cbdd88c4743f7b09e: No bootstrap required, opened a new log
I20260926 00:59:33.349433 3460 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79d75f67ca8c4b3cbdd88c4743f7b09e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79d75f67ca8c4b3cbdd88c4743f7b09e" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 34963 } }
I20260926 00:59:33.349632 3460 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79d75f67ca8c4b3cbdd88c4743f7b09e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 00:59:33.349669 3460 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79d75f67ca8c4b3cbdd88c4743f7b09e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79d75f67ca8c4b3cbdd88c4743f7b09e, State: Initialized, Role: FOLLOWER
I20260926 00:59:33.349804 3460 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79d75f67ca8c4b3cbdd88c4743f7b09e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79d75f67ca8c4b3cbdd88c4743f7b09e" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 34963 } }
I20260926 00:59:33.349889 3460 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79d75f67ca8c4b3cbdd88c4743f7b09e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 00:59:33.349920 3460 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79d75f67ca8c4b3cbdd88c4743f7b09e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 00:59:33.349969 3460 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79d75f67ca8c4b3cbdd88c4743f7b09e [term 0 FOLLOWER]: Advancing to term 1
I20260926 00:59:33.351301 3460 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79d75f67ca8c4b3cbdd88c4743f7b09e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79d75f67ca8c4b3cbdd88c4743f7b09e" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 34963 } }
I20260926 00:59:33.351429 3460 leader_election.cc:302] T 00000000000000000000000000000000 P 79d75f67ca8c4b3cbdd88c4743f7b09e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79d75f67ca8c4b3cbdd88c4743f7b09e; no voters:
I20260926 00:59:33.351614 3460 leader_election.cc:288] T 00000000000000000000000000000000 P 79d75f67ca8c4b3cbdd88c4743f7b09e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 00:59:33.351651 3465 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79d75f67ca8c4b3cbdd88c4743f7b09e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 00:59:33.351832 3465 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79d75f67ca8c4b3cbdd88c4743f7b09e [term 1 LEADER]: Becoming Leader. State: Replica: 79d75f67ca8c4b3cbdd88c4743f7b09e, State: Running, Role: LEADER
I20260926 00:59:33.351883 3460 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79d75f67ca8c4b3cbdd88c4743f7b09e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 00:59:33.351960 3465 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79d75f67ca8c4b3cbdd88c4743f7b09e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79d75f67ca8c4b3cbdd88c4743f7b09e" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 34963 } }
I20260926 00:59:33.352387 3466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79d75f67ca8c4b3cbdd88c4743f7b09e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79d75f67ca8c4b3cbdd88c4743f7b09e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79d75f67ca8c4b3cbdd88c4743f7b09e" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 34963 } } }
I20260926 00:59:33.352383 3465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79d75f67ca8c4b3cbdd88c4743f7b09e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79d75f67ca8c4b3cbdd88c4743f7b09e. Latest consensus state: current_term: 1 leader_uuid: "79d75f67ca8c4b3cbdd88c4743f7b09e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79d75f67ca8c4b3cbdd88c4743f7b09e" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 34963 } } }
I20260926 00:59:33.352476 3466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79d75f67ca8c4b3cbdd88c4743f7b09e [sys.catalog]: This master's current role is: LEADER
I20260926 00:59:33.352481 3465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79d75f67ca8c4b3cbdd88c4743f7b09e [sys.catalog]: This master's current role is: LEADER
I20260926 00:59:33.352734 3473 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 00:59:33.353308 3473 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 00:59:33.355005 3473 catalog_manager.cc:1347] Generated new cluster ID: c65f9b4a34d44413b0545a7e1d0aff41
I20260926 00:59:33.355057 3473 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 00:59:33 dist-test-slave-7v10 krb5kdc[3379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 00:59:33.365075 3473 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 00:59:33.365773 3473 catalog_manager.cc:1504] Loading token signing keys...
I20260926 00:59:33.366626 3464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:33.347853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43584 (local address 127.3.75.62:34963)
0926 00:59:33.348144 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:33.348151 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:33.348233 (+ 82us) server_negotiation.cc:407] Connection header received
0926 00:59:33.350802 (+ 2569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:33.350808 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:33.350850 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:33.351006 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:33.353404 (+ 2398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:33.354780 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:33.356307 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:33.356887 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:33.356993 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:33.361739 (+ 4746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:33.361742 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:33.361743 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 00:59:33.361754 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:33.364475 (+ 2721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:33.365008 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:33.365011 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:33.365011 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:33.365170 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:33.365438 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:33.365439 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:33.365439 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:33.365735 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:33.365924 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:33.366175 (+ 251us) server_negotiation.cc:299] Negotiation successful
0926 00:59:33.366301 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":249,"thread_start_us":75,"threads_started":1}
I20260926 00:59:33.370429 3473 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79d75f67ca8c4b3cbdd88c4743f7b09e: Generated new TSK 0
I20260926 00:59:33.370607 3473 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.75.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab.
Entry for principal kudu/127.3.75.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab.
I20260926 00:59:33.397487 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.1:0
--local_ip_for_outbound_sockets=127.3.75.1
--webserver_interface=127.3.75.1
--webserver_port=0
--tserver_master_addrs=127.3.75.62:34963
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab
--principal=kudu/127.3.75.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:42701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 00:59:33.476073 3486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 00:59:33.476280 3486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 00:59:33.476316 3486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 00:59:33.477382 3486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 00:59:33.477444 3486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 00:59:33.477489 3486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.1
I20260926 00:59:33.478989 3486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:42701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab
--principal=kudu/127.3.75.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.75.1
--webserver_port=0
--tserver_master_addrs=127.3.75.62:34963
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.1
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 00:59:33.479240 3486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 00:59:33.479470 3486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 00:59:33.485509 3492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:33.485679 3494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:33.485534 3491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 00:59:33.485792 3486 server_base.cc:1034] running on GCE node
I20260926 00:59:33.486003 3486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 00:59:33.486255 3486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 00:59:33.487412 3486 hybrid_clock.cc:648] HybridClock initialized: now 1790384373487395 us; error 36 us; skew 500 ppm
Sep 26 00:59:33 dist-test-slave-7v10 krb5kdc[3379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384373, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 00:59:33.489795 3486 init.cc:377] Logged in from keytab as kudu/127.3.75.1@KRBTEST.COM (short username kudu)
I20260926 00:59:33.490224 3486 webserver.cc:460] Webserver started at http://127.3.75.1:37353/ using document root <none> and password file <none>
I20260926 00:59:33.490463 3486 fs_manager.cc:359] Metadata directory not provided
I20260926 00:59:33.490530 3486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 00:59:33.490691 3486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 00:59:33.491816 3486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/ts-0/data/instance:
uuid: "e9921e4359e645e1bd842b9b4c0f5ab9"
format_stamp: "Formatted at 2026-09-26 00:59:33 on dist-test-slave-7v10"
server_key: "cd91454389ebd07701f2e921cad7eea3"
server_key_iv: "8548aff4bfad3e74d15af31dc81e8e8b"
server_key_version: "encryptionkey@0"
I20260926 00:59:33.492177 3486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/ts-0/wal/instance:
uuid: "e9921e4359e645e1bd842b9b4c0f5ab9"
format_stamp: "Formatted at 2026-09-26 00:59:33 on dist-test-slave-7v10"
server_key: "cd91454389ebd07701f2e921cad7eea3"
server_key_iv: "8548aff4bfad3e74d15af31dc81e8e8b"
server_key_version: "encryptionkey@0"
I20260926 00:59:33.493474 3486 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 00:59:33.494243 3501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:33.494424 3486 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 00:59:33.494520 3486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/ts-0/wal
uuid: "e9921e4359e645e1bd842b9b4c0f5ab9"
format_stamp: "Formatted at 2026-09-26 00:59:33 on dist-test-slave-7v10"
server_key: "cd91454389ebd07701f2e921cad7eea3"
server_key_iv: "8548aff4bfad3e74d15af31dc81e8e8b"
server_key_version: "encryptionkey@0"
I20260926 00:59:33.494606 3486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 00:59:33.518138 3486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 00:59:33.520044 3486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 00:59:33.520218 3486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 00:59:33.520452 3486 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 00:59:33.520766 3486 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 00:59:33.520824 3486 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:33.520865 3486 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 00:59:33.520895 3486 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:33.526726 3486 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.1:36749
I20260926 00:59:33.526798 3614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.1:36749 every 8 connection(s)
I20260926 00:59:33.527135 3486 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
I20260926 00:59:33.532500 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 3486
I20260926 00:59:33.532581 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/ts-0/wal/instance
I20260926 00:59:33.532771 3372 external_mini_cluster.cc:1428] Setting key e7bb6f69a3c1fa5d2bd8c30be0fdc489
Sep 26 00:59:33 dist-test-slave-7v10 krb5kdc[3379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384373, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.1@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 00:59:33.536770 3464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:33.527796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:44125 (local address 127.3.75.62:34963)
0926 00:59:33.527839 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:33.527841 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:33.528039 (+ 198us) server_negotiation.cc:407] Connection header received
0926 00:59:33.528575 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:33.528576 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:33.528589 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:33.528648 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:33.529673 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:33.530184 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:33.531304 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:33.531524 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:33.531596 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:33.533808 (+ 2212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:33.533809 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:33.533810 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 00:59:33.533816 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:33.535229 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:33.535729 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:33.535730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:33.535730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:33.535778 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:33.536059 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:33.536060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:33.536061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:33.536231 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:33.536295 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:33.536630 (+ 335us) server_negotiation.cc:299] Negotiation successful
0926 00:59:33.536707 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 00:59:33.537039 3617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:33.527776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34963 (local address 127.3.75.1:44125)
0926 00:59:33.527916 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:33.527955 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:33.528459 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:33.528718 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:33.528722 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:33.529371 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0926 00:59:33.529558 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:33.529563 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:33.530281 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:33.530282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:33.531213 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:33.531216 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:33.531641 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:33.531642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:33.531765 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:33.532184 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 00:59:33.532192 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 00:59:33.533716 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 00:59:33.535311 (+ 1595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 00:59:33.535317 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 00:59:33.535320 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 00:59:33.535646 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 00:59:33.535844 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 00:59:33.535845 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 00:59:33.535845 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 00:59:33.535986 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 00:59:33.536326 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 00:59:33.536332 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 00:59:33.536490 (+ 158us) client_negotiation.cc:769] Sending connection context
0926 00:59:33.536598 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 00:59:33.536771 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":72,"threads_started":1}
I20260926 00:59:33.537568 3615 heartbeater.cc:347] Connected to a master server at 127.3.75.62:34963
I20260926 00:59:33.537700 3615 heartbeater.cc:464] Registering TS with master...
I20260926 00:59:33.537870 3615 heartbeater.cc:511] Master 127.3.75.62:34963 requested a full tablet report, sending...
I20260926 00:59:33.538480 3424 ts_manager.cc:194] Registered new tserver with Master: e9921e4359e645e1bd842b9b4c0f5ab9 (127.3.75.1:36749)
I20260926 00:59:33.539523 3424 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.75.1@KRBTEST.COM'} at 127.3.75.1:44125
WARNING: no policy specified for kudu/127.3.75.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab.
Entry for principal kudu/127.3.75.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab.
I20260926 00:59:33.558036 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.2:0
--local_ip_for_outbound_sockets=127.3.75.2
--webserver_interface=127.3.75.2
--webserver_port=0
--tserver_master_addrs=127.3.75.62:34963
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab
--principal=kudu/127.3.75.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:42701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 00:59:33.637266 3620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 00:59:33.637454 3620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 00:59:33.637476 3620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 00:59:33.638563 3620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 00:59:33.638609 3620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 00:59:33.638661 3620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.2
I20260926 00:59:33.641168 3620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:42701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab
--principal=kudu/127.3.75.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.75.2
--webserver_port=0
--tserver_master_addrs=127.3.75.62:34963
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.2
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 00:59:33.641445 3620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 00:59:33.641729 3620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 00:59:33.647710 3625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:33.647737 3626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:33.648134 3628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 00:59:33.648186 3620 server_base.cc:1034] running on GCE node
I20260926 00:59:33.648360 3620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 00:59:33.648545 3620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 00:59:33.649677 3620 hybrid_clock.cc:648] HybridClock initialized: now 1790384373649668 us; error 26 us; skew 500 ppm
Sep 26 00:59:33 dist-test-slave-7v10 krb5kdc[3379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384373, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 00:59:33.652230 3620 init.cc:377] Logged in from keytab as kudu/127.3.75.2@KRBTEST.COM (short username kudu)
I20260926 00:59:33.652694 3620 webserver.cc:460] Webserver started at http://127.3.75.2:38719/ using document root <none> and password file <none>
I20260926 00:59:33.652873 3620 fs_manager.cc:359] Metadata directory not provided
I20260926 00:59:33.652912 3620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 00:59:33.653034 3620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 00:59:33.654044 3620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/ts-1/data/instance:
uuid: "c71c4cc69949406aa5d4da44554f0044"
format_stamp: "Formatted at 2026-09-26 00:59:33 on dist-test-slave-7v10"
server_key: "e94f74067d506aa8734315e43f63496b"
server_key_iv: "3b8cc952c2dfff73dcdb5de5265740de"
server_key_version: "encryptionkey@0"
I20260926 00:59:33.654356 3620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/ts-1/wal/instance:
uuid: "c71c4cc69949406aa5d4da44554f0044"
format_stamp: "Formatted at 2026-09-26 00:59:33 on dist-test-slave-7v10"
server_key: "e94f74067d506aa8734315e43f63496b"
server_key_iv: "3b8cc952c2dfff73dcdb5de5265740de"
server_key_version: "encryptionkey@0"
I20260926 00:59:33.655812 3620 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 00:59:33.656512 3635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:33.656687 3620 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:33.656791 3620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/ts-1/wal
uuid: "c71c4cc69949406aa5d4da44554f0044"
format_stamp: "Formatted at 2026-09-26 00:59:33 on dist-test-slave-7v10"
server_key: "e94f74067d506aa8734315e43f63496b"
server_key_iv: "3b8cc952c2dfff73dcdb5de5265740de"
server_key_version: "encryptionkey@0"
I20260926 00:59:33.656865 3620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 00:59:33.671912 3620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 00:59:33.674018 3620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 00:59:33.674197 3620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 00:59:33.674451 3620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 00:59:33.674777 3620 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 00:59:33.674831 3620 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:33.674870 3620 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 00:59:33.674919 3620 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:33.680950 3620 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.2:40779
I20260926 00:59:33.681005 3748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.2:40779 every 8 connection(s)
I20260926 00:59:33.681365 3620 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
I20260926 00:59:33.682942 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 3620
I20260926 00:59:33.683084 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/ts-1/wal/instance
I20260926 00:59:33.683267 3372 external_mini_cluster.cc:1428] Setting key c3655e2c577a408259693fce15496341
Sep 26 00:59:33 dist-test-slave-7v10 krb5kdc[3379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384373, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.2@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 00:59:33.692032 3464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:33.682013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:58053 (local address 127.3.75.62:34963)
0926 00:59:33.682086 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:33.682089 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:33.682317 (+ 228us) server_negotiation.cc:407] Connection header received
0926 00:59:33.683050 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:33.683052 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:33.683067 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:33.683137 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:33.684520 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:33.685015 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:33.686218 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:33.686398 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:33.686455 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:33.688924 (+ 2469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:33.688926 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:33.688927 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 00:59:33.688935 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:33.690653 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:33.691104 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:33.691105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:33.691105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:33.691168 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:33.691423 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:33.691423 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:33.691424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:33.691534 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:33.691610 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:33.691866 (+ 256us) server_negotiation.cc:299] Negotiation successful
0926 00:59:33.691946 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 00:59:33.692204 3751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:33.682008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34963 (local address 127.3.75.2:58053)
0926 00:59:33.682186 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:33.682223 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:33.682848 (+ 625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:33.683233 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:33.683239 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:33.684095 (+ 856us) client_negotiation.cc:189] Negotiated authn=SASL
0926 00:59:33.684409 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:33.684415 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:33.685100 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:33.685101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:33.686148 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:33.686151 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:33.686494 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:33.686495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:33.686656 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:33.687189 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 00:59:33.687200 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 00:59:33.688830 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 00:59:33.690734 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 00:59:33.690740 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 00:59:33.690742 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 00:59:33.691026 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 00:59:33.691247 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 00:59:33.691248 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 00:59:33.691248 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 00:59:33.691349 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 00:59:33.691604 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 00:59:33.691610 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 00:59:33.691739 (+ 129us) client_negotiation.cc:769] Sending connection context
0926 00:59:33.691833 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 00:59:33.691986 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":88,"threads_started":1}
WARNING: no policy specified for kudu/127.3.75.3@KRBTEST.COM; defaulting to no policy
I20260926 00:59:33.692597 3749 heartbeater.cc:347] Connected to a master server at 127.3.75.62:34963
I20260926 00:59:33.692708 3749 heartbeater.cc:464] Registering TS with master...
I20260926 00:59:33.692874 3749 heartbeater.cc:511] Master 127.3.75.62:34963 requested a full tablet report, sending...
I20260926 00:59:33.693244 3424 ts_manager.cc:194] Registered new tserver with Master: c71c4cc69949406aa5d4da44554f0044 (127.3.75.2:40779)
I20260926 00:59:33.693632 3424 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.75.2@KRBTEST.COM'} at 127.3.75.2:58053
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab.
Entry for principal kudu/127.3.75.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab.
I20260926 00:59:33.709489 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.3:0
--local_ip_for_outbound_sockets=127.3.75.3
--webserver_interface=127.3.75.3
--webserver_port=0
--tserver_master_addrs=127.3.75.62:34963
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab
--principal=kudu/127.3.75.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:42701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 00:59:33.790208 3754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 00:59:33.790452 3754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 00:59:33.790491 3754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 00:59:33.791647 3754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 00:59:33.791719 3754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 00:59:33.791764 3754 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.3
I20260926 00:59:33.793181 3754 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:42701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab
--principal=kudu/127.3.75.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.75.3
--webserver_port=0
--tserver_master_addrs=127.3.75.62:34963
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3754
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.3
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 00:59:33.793478 3754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 00:59:33.793730 3754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 00:59:33.798641 3759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:33.798692 3760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:33.798645 3762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 00:59:33.798954 3754 server_base.cc:1034] running on GCE node
I20260926 00:59:33.799165 3754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 00:59:33.799415 3754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 00:59:33.800575 3754 hybrid_clock.cc:648] HybridClock initialized: now 1790384373800562 us; error 44 us; skew 500 ppm
Sep 26 00:59:33 dist-test-slave-7v10 krb5kdc[3379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384373, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 00:59:33.803058 3754 init.cc:377] Logged in from keytab as kudu/127.3.75.3@KRBTEST.COM (short username kudu)
I20260926 00:59:33.803485 3754 webserver.cc:460] Webserver started at http://127.3.75.3:45859/ using document root <none> and password file <none>
I20260926 00:59:33.803707 3754 fs_manager.cc:359] Metadata directory not provided
I20260926 00:59:33.803778 3754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 00:59:33.803925 3754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 00:59:33.804960 3754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/ts-2/data/instance:
uuid: "c60444bc05f74c4488ea1edf3d893b97"
format_stamp: "Formatted at 2026-09-26 00:59:33 on dist-test-slave-7v10"
server_key: "00cfd3b8434ca7237493c6c997e3a5a8"
server_key_iv: "e1e464a79a6cf1fd1ecee896f395853c"
server_key_version: "encryptionkey@0"
I20260926 00:59:33.805377 3754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/ts-2/wal/instance:
uuid: "c60444bc05f74c4488ea1edf3d893b97"
format_stamp: "Formatted at 2026-09-26 00:59:33 on dist-test-slave-7v10"
server_key: "00cfd3b8434ca7237493c6c997e3a5a8"
server_key_iv: "e1e464a79a6cf1fd1ecee896f395853c"
server_key_version: "encryptionkey@0"
I20260926 00:59:33.806669 3754 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 00:59:33.807446 3769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:33.807624 3754 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:33.807722 3754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/ts-2/wal
uuid: "c60444bc05f74c4488ea1edf3d893b97"
format_stamp: "Formatted at 2026-09-26 00:59:33 on dist-test-slave-7v10"
server_key: "00cfd3b8434ca7237493c6c997e3a5a8"
server_key_iv: "e1e464a79a6cf1fd1ecee896f395853c"
server_key_version: "encryptionkey@0"
I20260926 00:59:33.807797 3754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 00:59:33.833806 3754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 00:59:33.836051 3754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 00:59:33.836221 3754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 00:59:33.836439 3754 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 00:59:33.836737 3754 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 00:59:33.836792 3754 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:33.836850 3754 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 00:59:33.836902 3754 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:33.842344 3754 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.3:43359
I20260926 00:59:33.842427 3882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.3:43359 every 8 connection(s)
I20260926 00:59:33.842736 3754 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
I20260926 00:59:33.844514 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 3754
I20260926 00:59:33.844635 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384372073861-3372-0/minicluster-data/ts-2/wal/instance
I20260926 00:59:33.844820 3372 external_mini_cluster.cc:1428] Setting key 2ae5f99269668d095eb9ece3bdc98f82
Sep 26 00:59:33 dist-test-slave-7v10 krb5kdc[3379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384373, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.3@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 00:59:33.851774 3464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:33.843496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.3:51561 (local address 127.3.75.62:34963)
0926 00:59:33.843576 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:33.843586 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:33.843794 (+ 208us) server_negotiation.cc:407] Connection header received
0926 00:59:33.844264 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:33.844265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:33.844275 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:33.844325 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:33.845593 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:33.846178 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:33.847213 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:33.847396 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:33.847453 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:33.849357 (+ 1904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:33.849358 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:33.849358 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 00:59:33.849365 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:33.850510 (+ 1145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:33.850851 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:33.850851 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:33.850851 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:33.850951 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:33.851170 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:33.851170 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:33.851171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:33.851267 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:33.851316 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:33.851593 (+ 277us) server_negotiation.cc:299] Negotiation successful
0926 00:59:33.851652 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 00:59:33.851945 3885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:33.843458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34963 (local address 127.3.75.3:51561)
0926 00:59:33.843663 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:33.843708 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:33.844180 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:33.844422 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:33.844427 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:33.845223 (+ 796us) client_negotiation.cc:189] Negotiated authn=SASL
0926 00:59:33.845485 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:33.845492 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:33.846256 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:33.846257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:33.847145 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:33.847148 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:33.847469 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:33.847469 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:33.847566 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:33.847977 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 00:59:33.847984 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 00:59:33.849275 (+ 1291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 00:59:33.850579 (+ 1304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 00:59:33.850584 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 00:59:33.850586 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 00:59:33.850787 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 00:59:33.851007 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 00:59:33.851008 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 00:59:33.851008 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 00:59:33.851102 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 00:59:33.851324 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 00:59:33.851328 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 00:59:33.851434 (+ 106us) client_negotiation.cc:769] Sending connection context
0926 00:59:33.851566 (+ 132us) client_negotiation.cc:240] Negotiation successful
0926 00:59:33.851688 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":106,"threads_started":1}
I20260926 00:59:33.852475 3883 heartbeater.cc:347] Connected to a master server at 127.3.75.62:34963
I20260926 00:59:33.852583 3883 heartbeater.cc:464] Registering TS with master...
I20260926 00:59:33.852774 3883 heartbeater.cc:511] Master 127.3.75.62:34963 requested a full tablet report, sending...
I20260926 00:59:33.853116 3424 ts_manager.cc:194] Registered new tserver with Master: c60444bc05f74c4488ea1edf3d893b97 (127.3.75.3:43359)
I20260926 00:59:33.853552 3424 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.75.3@KRBTEST.COM'} at 127.3.75.3:51561
I20260926 00:59:33.856510 3372 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 00:59:33 dist-test-slave-7v10 krb5kdc[3379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384373, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 00:59:33 dist-test-slave-7v10 krb5kdc[3379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384373, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.75.1@KRBTEST.COM
I20260926 00:59:33.878870 3890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:33.871528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33692 (local address 127.3.75.1:36749)
0926 00:59:33.871708 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:33.871712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:33.871722 (+ 10us) server_negotiation.cc:407] Connection header received
0926 00:59:33.871746 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:33.871748 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:33.871760 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:33.871825 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:33.872311 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:33.873019 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:33.873738 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:33.874039 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:33.874101 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:33.876390 (+ 2289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:33.876398 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:33.876398 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 00:59:33.876405 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:33.877890 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:33.878199 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:33.878201 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:33.878202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:33.878252 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:33.878409 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:33.878410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:33.878410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:33.878523 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:33.878619 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:33.878685 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 00:59:33.878742 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":87,"threads_started":1}
W20260926 00:59:33.879277 3529 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33692
Sep 26 00:59:33 dist-test-slave-7v10 krb5kdc[3379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 00:59:33 dist-test-slave-7v10 krb5kdc[3379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.75.1@KRBTEST.COM
I20260926 00:59:33.900743 3890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:33.893720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33696 (local address 127.3.75.1:36749)
0926 00:59:33.893772 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:33.893774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:33.893862 (+ 88us) server_negotiation.cc:407] Connection header received
0926 00:59:33.893896 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:33.893897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:33.893909 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:33.893950 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:33.894502 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:33.895054 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:33.895746 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:33.895940 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:33.895990 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:33.898251 (+ 2261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:33.898252 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:33.898252 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 00:59:33.898258 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:33.899557 (+ 1299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:33.899860 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:33.899861 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:33.899861 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:33.899910 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:33.900152 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:33.900153 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:33.900153 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:33.900273 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:33.900413 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:33.900558 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 00:59:33.900657 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 00:59:33.901268 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 3486
I20260926 00:59:33.906461 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 3620
I20260926 00:59:33.911949 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 3754
I20260926 00:59:33.916481 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 3392
2026-09-26T00:59:33Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1851 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 00:59:33 dist-test-slave-7v10 krb5kdc[3897](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 00:59:33 dist-test-slave-7v10 krb5kdc[3897](info): set up 2 sockets
Sep 26 00:59:33 dist-test-slave-7v10 krb5kdc[3897](info): commencing operation
krb5kdc: starting...
W20260926 00:59:34.957422 3372 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Sep 26 00:59:35 dist-test-slave-7v10 krb5kdc[3897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T00:59:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T00:59:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.75.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab.
Entry for principal kudu/127.3.75.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab.
I20260926 00:59:35.071677 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:38659
--webserver_interface=127.3.75.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:45227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.75.62:38659
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 00:59:35.151294 3910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 00:59:35.151518 3910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 00:59:35.151540 3910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 00:59:35.152622 3910 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 00:59:35.152657 3910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 00:59:35.152671 3910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 00:59:35.152683 3910 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 00:59:35.152695 3910 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 00:59:35.154057 3910 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:45227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.75.62:38659
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:38659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3910
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 00:59:35.154228 3910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 00:59:35.154394 3910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 00:59:35.159073 3918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:35.159122 3916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:35.159091 3915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 00:59:35.159355 3910 server_base.cc:1034] running on GCE node
I20260926 00:59:35.159598 3910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 00:59:35.159834 3910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 00:59:35.160969 3910 hybrid_clock.cc:648] HybridClock initialized: now 1790384375160966 us; error 36 us; skew 500 ppm
Sep 26 00:59:35 dist-test-slave-7v10 krb5kdc[3897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384375, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 00:59:35.163357 3910 init.cc:377] Logged in from keytab as kudu/127.3.75.62@KRBTEST.COM (short username kudu)
I20260926 00:59:35.163825 3910 webserver.cc:460] Webserver started at http://127.3.75.62:36813/ using document root <none> and password file <none>
I20260926 00:59:35.164067 3910 fs_manager.cc:359] Metadata directory not provided
I20260926 00:59:35.164137 3910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 00:59:35.164294 3910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 00:59:35.165313 3910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/master-0/data/instance:
uuid: "fa1944a0c86441d0830280614c82bf6d"
format_stamp: "Formatted at 2026-09-26 00:59:35 on dist-test-slave-7v10"
server_key: "4244e1f194e2776138146b9bf22d4bf9"
server_key_iv: "80a97a0ebb125a3b7497368a397e1fc4"
server_key_version: "encryptionkey@0"
I20260926 00:59:35.165762 3910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/master-0/wal/instance:
uuid: "fa1944a0c86441d0830280614c82bf6d"
format_stamp: "Formatted at 2026-09-26 00:59:35 on dist-test-slave-7v10"
server_key: "4244e1f194e2776138146b9bf22d4bf9"
server_key_iv: "80a97a0ebb125a3b7497368a397e1fc4"
server_key_version: "encryptionkey@0"
I20260926 00:59:35.167122 3910 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 00:59:35.167847 3925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:35.168018 3910 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 00:59:35.168124 3910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/master-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/master-0/wal
uuid: "fa1944a0c86441d0830280614c82bf6d"
format_stamp: "Formatted at 2026-09-26 00:59:35 on dist-test-slave-7v10"
server_key: "4244e1f194e2776138146b9bf22d4bf9"
server_key_iv: "80a97a0ebb125a3b7497368a397e1fc4"
server_key_version: "encryptionkey@0"
I20260926 00:59:35.168203 3910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 00:59:35.183108 3910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 00:59:35.184768 3910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 00:59:35.184926 3910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 00:59:35.188828 3910 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.62:38659
I20260926 00:59:35.188860 3977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.62:38659 every 8 connection(s)
I20260926 00:59:35.189195 3910 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
I20260926 00:59:35.189999 3978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 00:59:35.192222 3978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa1944a0c86441d0830280614c82bf6d: Bootstrap starting.
I20260926 00:59:35.192725 3978 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa1944a0c86441d0830280614c82bf6d: Neither blocks nor log segments found. Creating new log.
I20260926 00:59:35.192957 3978 log.cc:824] T 00000000000000000000000000000000 P fa1944a0c86441d0830280614c82bf6d: Log is configured to *not* fsync() on all Append() calls
I20260926 00:59:35.193547 3978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa1944a0c86441d0830280614c82bf6d: No bootstrap required, opened a new log
I20260926 00:59:35.195132 3978 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa1944a0c86441d0830280614c82bf6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa1944a0c86441d0830280614c82bf6d" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 38659 } }
I20260926 00:59:35.195262 3978 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa1944a0c86441d0830280614c82bf6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 00:59:35.195283 3978 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa1944a0c86441d0830280614c82bf6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa1944a0c86441d0830280614c82bf6d, State: Initialized, Role: FOLLOWER
I20260926 00:59:35.195411 3978 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa1944a0c86441d0830280614c82bf6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa1944a0c86441d0830280614c82bf6d" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 38659 } }
I20260926 00:59:35.195497 3978 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa1944a0c86441d0830280614c82bf6d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 00:59:35.195520 3978 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa1944a0c86441d0830280614c82bf6d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 00:59:35.195550 3978 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa1944a0c86441d0830280614c82bf6d [term 0 FOLLOWER]: Advancing to term 1
I20260926 00:59:35.196002 3978 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa1944a0c86441d0830280614c82bf6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa1944a0c86441d0830280614c82bf6d" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 38659 } }
I20260926 00:59:35.196079 3978 leader_election.cc:302] T 00000000000000000000000000000000 P fa1944a0c86441d0830280614c82bf6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa1944a0c86441d0830280614c82bf6d; no voters:
I20260926 00:59:35.196209 3978 leader_election.cc:288] T 00000000000000000000000000000000 P fa1944a0c86441d0830280614c82bf6d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 00:59:35.196277 3981 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa1944a0c86441d0830280614c82bf6d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 00:59:35.196400 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 3910
I20260926 00:59:35.196485 3978 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa1944a0c86441d0830280614c82bf6d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 00:59:35.196496 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/master-0/wal/instance
I20260926 00:59:35.196637 3981 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa1944a0c86441d0830280614c82bf6d [term 1 LEADER]: Becoming Leader. State: Replica: fa1944a0c86441d0830280614c82bf6d, State: Running, Role: LEADER
I20260926 00:59:35.196684 3372 external_mini_cluster.cc:1428] Setting key 686ecbdbbec85d4b123e41b1d80761d3
I20260926 00:59:35.196728 3981 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa1944a0c86441d0830280614c82bf6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa1944a0c86441d0830280614c82bf6d" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 38659 } }
I20260926 00:59:35.197121 3982 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa1944a0c86441d0830280614c82bf6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa1944a0c86441d0830280614c82bf6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa1944a0c86441d0830280614c82bf6d" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 38659 } } }
I20260926 00:59:35.197212 3982 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa1944a0c86441d0830280614c82bf6d [sys.catalog]: This master's current role is: LEADER
I20260926 00:59:35.197417 3983 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa1944a0c86441d0830280614c82bf6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa1944a0c86441d0830280614c82bf6d. Latest consensus state: current_term: 1 leader_uuid: "fa1944a0c86441d0830280614c82bf6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa1944a0c86441d0830280614c82bf6d" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 38659 } } }
I20260926 00:59:35.197556 3983 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa1944a0c86441d0830280614c82bf6d [sys.catalog]: This master's current role is: LEADER
I20260926 00:59:35.197960 3992 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 00:59:35.198451 3992 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 00:59:35.199923 3992 catalog_manager.cc:1347] Generated new cluster ID: 6375831578af41e69c26e7703462bd1a
I20260926 00:59:35.199973 3992 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 00:59:35 dist-test-slave-7v10 krb5kdc[3897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 00:59:35.207372 3991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:35.197183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40316 (local address 127.3.75.62:38659)
0926 00:59:35.197383 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:35.197392 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:35.197421 (+ 29us) server_negotiation.cc:407] Connection header received
0926 00:59:35.197532 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:35.197538 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:35.197606 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:35.197775 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:35.198683 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:35.199883 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:35.200729 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:35.201142 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:35.201261 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:35.203550 (+ 2289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:35.203554 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:35.203554 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 00:59:35.203565 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:35.205737 (+ 2172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:35.206138 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:35.206141 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:35.206141 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:35.206225 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:35.206453 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:35.206454 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:35.206455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:35.206678 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:35.206875 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:35.207016 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 00:59:35.207171 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":74,"threads_started":1}
I20260926 00:59:35.208022 3992 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 00:59:35.208675 3992 catalog_manager.cc:1504] Loading token signing keys...
I20260926 00:59:35.214377 3992 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa1944a0c86441d0830280614c82bf6d: Generated new TSK 0
I20260926 00:59:35.214565 3992 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.75.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab.
Entry for principal kudu/127.3.75.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab.
I20260926 00:59:35.245832 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.1:0
--local_ip_for_outbound_sockets=127.3.75.1
--webserver_interface=127.3.75.1
--webserver_port=0
--tserver_master_addrs=127.3.75.62:38659
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab
--principal=kudu/127.3.75.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:45227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 00:59:35.341857 4005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 00:59:35.342056 4005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 00:59:35.342079 4005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 00:59:35.343299 4005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 00:59:35.343339 4005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 00:59:35.343366 4005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.1
I20260926 00:59:35.344856 4005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:45227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab
--principal=kudu/127.3.75.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.75.1
--webserver_port=0
--tserver_master_addrs=127.3.75.62:38659
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.1
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 00:59:35.345100 4005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 00:59:35.345371 4005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 00:59:35.351466 4010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:35.351560 4013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:35.351466 4011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 00:59:35.351835 4005 server_base.cc:1034] running on GCE node
I20260926 00:59:35.352066 4005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 00:59:35.352311 4005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 00:59:35.353472 4005 hybrid_clock.cc:648] HybridClock initialized: now 1790384375353453 us; error 37 us; skew 500 ppm
Sep 26 00:59:35 dist-test-slave-7v10 krb5kdc[3897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384375, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 00:59:35.356149 4005 init.cc:377] Logged in from keytab as kudu/127.3.75.1@KRBTEST.COM (short username kudu)
I20260926 00:59:35.356624 4005 webserver.cc:460] Webserver started at http://127.3.75.1:34007/ using document root <none> and password file <none>
I20260926 00:59:35.356877 4005 fs_manager.cc:359] Metadata directory not provided
I20260926 00:59:35.356922 4005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 00:59:35.357064 4005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 00:59:35.358167 4005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/ts-0/data/instance:
uuid: "ad79ef498e8e4076ac30220e9228c3cf"
format_stamp: "Formatted at 2026-09-26 00:59:35 on dist-test-slave-7v10"
server_key: "97e8081137ad641d9b0b4e1b3fe04884"
server_key_iv: "16d1f6c60cf1769b4d38d2d8668ebb02"
server_key_version: "encryptionkey@0"
I20260926 00:59:35.358512 4005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/ts-0/wal/instance:
uuid: "ad79ef498e8e4076ac30220e9228c3cf"
format_stamp: "Formatted at 2026-09-26 00:59:35 on dist-test-slave-7v10"
server_key: "97e8081137ad641d9b0b4e1b3fe04884"
server_key_iv: "16d1f6c60cf1769b4d38d2d8668ebb02"
server_key_version: "encryptionkey@0"
I20260926 00:59:35.359930 4005 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 00:59:35.360646 4020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:35.360824 4005 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 00:59:35.360944 4005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/ts-0/wal
uuid: "ad79ef498e8e4076ac30220e9228c3cf"
format_stamp: "Formatted at 2026-09-26 00:59:35 on dist-test-slave-7v10"
server_key: "97e8081137ad641d9b0b4e1b3fe04884"
server_key_iv: "16d1f6c60cf1769b4d38d2d8668ebb02"
server_key_version: "encryptionkey@0"
I20260926 00:59:35.361022 4005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 00:59:35.372443 4005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 00:59:35.374684 4005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 00:59:35.374838 4005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 00:59:35.375116 4005 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 00:59:35.375406 4005 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 00:59:35.375458 4005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:35.375500 4005 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 00:59:35.375526 4005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:35.381249 4005 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.1:34373
I20260926 00:59:35.381314 4133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.1:34373 every 8 connection(s)
I20260926 00:59:35.381601 4005 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
Sep 26 00:59:35 dist-test-slave-7v10 krb5kdc[3897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384375, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.1@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 00:59:35.390241 3991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:35.382246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:37653 (local address 127.3.75.62:38659)
0926 00:59:35.382315 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:35.382317 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:35.382526 (+ 209us) server_negotiation.cc:407] Connection header received
0926 00:59:35.383538 (+ 1012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:35.383539 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:35.383551 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:35.383606 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:35.384198 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:35.384687 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:35.385629 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:35.385818 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:35.385864 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:35.388016 (+ 2152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:35.388017 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:35.388017 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 00:59:35.388023 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:35.389173 (+ 1150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:35.389503 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:35.389504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:35.389505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:35.389548 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:35.389745 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:35.389746 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:35.389747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:35.389855 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:35.389902 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:35.390120 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 00:59:35.390179 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 00:59:35.390466 4136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:35.382245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:38659 (local address 127.3.75.1:37653)
0926 00:59:35.382404 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:35.382438 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:35.383445 (+ 1007us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:35.383689 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:35.383692 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:35.383933 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0926 00:59:35.384119 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:35.384123 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:35.384762 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:35.384762 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:35.385542 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:35.385544 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:35.385885 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:35.385886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:35.385982 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:35.386492 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 00:59:35.386500 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 00:59:35.387927 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 00:59:35.389239 (+ 1312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 00:59:35.389245 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 00:59:35.389246 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 00:59:35.389443 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 00:59:35.389606 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 00:59:35.389607 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 00:59:35.389607 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 00:59:35.389696 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 00:59:35.389907 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 00:59:35.389910 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 00:59:35.390015 (+ 105us) client_negotiation.cc:769] Sending connection context
0926 00:59:35.390095 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 00:59:35.390210 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":92,"threads_started":1}
I20260926 00:59:35.391024 4134 heartbeater.cc:347] Connected to a master server at 127.3.75.62:38659
I20260926 00:59:35.391116 4134 heartbeater.cc:464] Registering TS with master...
I20260926 00:59:35.391247 4134 heartbeater.cc:511] Master 127.3.75.62:38659 requested a full tablet report, sending...
I20260926 00:59:35.391474 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 4005
I20260926 00:59:35.391592 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/ts-0/wal/instance
I20260926 00:59:35.391753 3372 external_mini_cluster.cc:1428] Setting key bdc2223b1d874e37b121643115ca62ae
I20260926 00:59:35.391808 3942 ts_manager.cc:194] Registered new tserver with Master: ad79ef498e8e4076ac30220e9228c3cf (127.3.75.1:34373)
I20260926 00:59:35.392577 3942 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.75.1@KRBTEST.COM'} at 127.3.75.1:37653
WARNING: no policy specified for kudu/127.3.75.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab.
Entry for principal kudu/127.3.75.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab.
I20260926 00:59:35.415194 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.2:0
--local_ip_for_outbound_sockets=127.3.75.2
--webserver_interface=127.3.75.2
--webserver_port=0
--tserver_master_addrs=127.3.75.62:38659
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab
--principal=kudu/127.3.75.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:45227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 00:59:35.493364 4139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 00:59:35.493605 4139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 00:59:35.493647 4139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 00:59:35.494791 4139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 00:59:35.494863 4139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 00:59:35.494939 4139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.2
I20260926 00:59:35.496619 4139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:45227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab
--principal=kudu/127.3.75.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.75.2
--webserver_port=0
--tserver_master_addrs=127.3.75.62:38659
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.2
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 00:59:35.496876 4139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 00:59:35.497134 4139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 00:59:35.502274 4147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:35.502336 4144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 00:59:35.502323 4139 server_base.cc:1034] running on GCE node
W20260926 00:59:35.502337 4145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 00:59:35.502753 4139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 00:59:35.503069 4139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 00:59:35.504299 4139 hybrid_clock.cc:648] HybridClock initialized: now 1790384375504270 us; error 45 us; skew 500 ppm
Sep 26 00:59:35 dist-test-slave-7v10 krb5kdc[3897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384375, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 00:59:35.506717 4139 init.cc:377] Logged in from keytab as kudu/127.3.75.2@KRBTEST.COM (short username kudu)
I20260926 00:59:35.507135 4139 webserver.cc:460] Webserver started at http://127.3.75.2:44151/ using document root <none> and password file <none>
I20260926 00:59:35.507318 4139 fs_manager.cc:359] Metadata directory not provided
I20260926 00:59:35.507356 4139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 00:59:35.507437 4139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 00:59:35.508378 4139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/ts-1/data/instance:
uuid: "a8b5b8b0bc2046b182164e8b521c28a8"
format_stamp: "Formatted at 2026-09-26 00:59:35 on dist-test-slave-7v10"
server_key: "06f5be6d51847d90b68b67bdd94e39a9"
server_key_iv: "731c6c477b2be8a3275a7b409dc1e57a"
server_key_version: "encryptionkey@0"
I20260926 00:59:35.508653 4139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/ts-1/wal/instance:
uuid: "a8b5b8b0bc2046b182164e8b521c28a8"
format_stamp: "Formatted at 2026-09-26 00:59:35 on dist-test-slave-7v10"
server_key: "06f5be6d51847d90b68b67bdd94e39a9"
server_key_iv: "731c6c477b2be8a3275a7b409dc1e57a"
server_key_version: "encryptionkey@0"
I20260926 00:59:35.509828 4139 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 00:59:35.510488 4154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:35.510666 4139 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 00:59:35.510730 4139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/ts-1/wal
uuid: "a8b5b8b0bc2046b182164e8b521c28a8"
format_stamp: "Formatted at 2026-09-26 00:59:35 on dist-test-slave-7v10"
server_key: "06f5be6d51847d90b68b67bdd94e39a9"
server_key_iv: "731c6c477b2be8a3275a7b409dc1e57a"
server_key_version: "encryptionkey@0"
I20260926 00:59:35.510815 4139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 00:59:35.535794 4139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 00:59:35.537658 4139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 00:59:35.537820 4139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 00:59:35.538069 4139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 00:59:35.538400 4139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 00:59:35.538455 4139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:35.538498 4139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 00:59:35.538527 4139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:35.544641 4139 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.2:38229
I20260926 00:59:35.544684 4267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.2:38229 every 8 connection(s)
I20260926 00:59:35.544994 4139 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
I20260926 00:59:35.549885 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 4139
I20260926 00:59:35.549986 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/ts-1/wal/instance
I20260926 00:59:35.550164 3372 external_mini_cluster.cc:1428] Setting key 2cdf94477bae57ba9ca14d97f3641383
Sep 26 00:59:35 dist-test-slave-7v10 krb5kdc[3897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384375, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.2@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 00:59:35.554575 3991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:35.545709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:32789 (local address 127.3.75.62:38659)
0926 00:59:35.545777 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:35.545779 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:35.545933 (+ 154us) server_negotiation.cc:407] Connection header received
0926 00:59:35.546897 (+ 964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:35.546898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:35.546913 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:35.546957 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:35.547602 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:35.548098 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:35.549130 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:35.549319 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:35.549420 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:35.551714 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:35.551716 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:35.551716 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 00:59:35.551724 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:35.553202 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:35.553634 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:35.553636 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:35.553636 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:35.553696 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:35.553936 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:35.553937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:35.553938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:35.554077 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:35.554154 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:35.554411 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 00:59:35.554488 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 00:59:35.554751 4270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:35.545662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:38659 (local address 127.3.75.2:32789)
0926 00:59:35.545811 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:35.545852 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:35.546748 (+ 896us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:35.547041 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:35.547045 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:35.547303 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0926 00:59:35.547488 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:35.547492 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:35.548191 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:35.548192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:35.549022 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:35.549024 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:35.549426 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:35.549427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:35.549549 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:35.550006 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 00:59:35.550016 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 00:59:35.551628 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 00:59:35.553315 (+ 1687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 00:59:35.553321 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 00:59:35.553323 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 00:59:35.553553 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 00:59:35.553762 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 00:59:35.553763 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 00:59:35.553763 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 00:59:35.553869 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 00:59:35.554146 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 00:59:35.554152 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 00:59:35.554287 (+ 135us) client_negotiation.cc:769] Sending connection context
0926 00:59:35.554390 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 00:59:35.554537 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
I20260926 00:59:35.555215 4268 heartbeater.cc:347] Connected to a master server at 127.3.75.62:38659
I20260926 00:59:35.555301 4268 heartbeater.cc:464] Registering TS with master...
I20260926 00:59:35.555502 4268 heartbeater.cc:511] Master 127.3.75.62:38659 requested a full tablet report, sending...
I20260926 00:59:35.555889 3942 ts_manager.cc:194] Registered new tserver with Master: a8b5b8b0bc2046b182164e8b521c28a8 (127.3.75.2:38229)
I20260926 00:59:35.556278 3942 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.75.2@KRBTEST.COM'} at 127.3.75.2:32789
WARNING: no policy specified for kudu/127.3.75.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab.
Entry for principal kudu/127.3.75.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab.
I20260926 00:59:35.573596 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.3:0
--local_ip_for_outbound_sockets=127.3.75.3
--webserver_interface=127.3.75.3
--webserver_port=0
--tserver_master_addrs=127.3.75.62:38659
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab
--principal=kudu/127.3.75.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:45227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 00:59:35.662932 4273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 00:59:35.663138 4273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 00:59:35.663162 4273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 00:59:35.664507 4273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 00:59:35.664546 4273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 00:59:35.664605 4273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.3
I20260926 00:59:35.666106 4273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:45227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab
--principal=kudu/127.3.75.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.75.3
--webserver_port=0
--tserver_master_addrs=127.3.75.62:38659
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.3
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 00:59:35.666332 4273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 00:59:35.666564 4273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 00:59:35.671551 4278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:35.671561 4279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:35.671578 4281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 00:59:35.671881 4273 server_base.cc:1034] running on GCE node
I20260926 00:59:35.672091 4273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 00:59:35.672269 4273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 00:59:35.673404 4273 hybrid_clock.cc:648] HybridClock initialized: now 1790384375673387 us; error 33 us; skew 500 ppm
Sep 26 00:59:35 dist-test-slave-7v10 krb5kdc[3897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384375, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 00:59:35.679858 4273 init.cc:377] Logged in from keytab as kudu/127.3.75.3@KRBTEST.COM (short username kudu)
I20260926 00:59:35.680244 4273 webserver.cc:460] Webserver started at http://127.3.75.3:45251/ using document root <none> and password file <none>
I20260926 00:59:35.680416 4273 fs_manager.cc:359] Metadata directory not provided
I20260926 00:59:35.680454 4273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 00:59:35.680536 4273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 00:59:35.681514 4273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/ts-2/data/instance:
uuid: "b340ff8635c54e6eb4ed36a2a8613563"
format_stamp: "Formatted at 2026-09-26 00:59:35 on dist-test-slave-7v10"
server_key: "2122e6c7919e422aed92d9ddd2112918"
server_key_iv: "be5a7ddaaedbe06ababb7faa8c100f2b"
server_key_version: "encryptionkey@0"
I20260926 00:59:35.681808 4273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/ts-2/wal/instance:
uuid: "b340ff8635c54e6eb4ed36a2a8613563"
format_stamp: "Formatted at 2026-09-26 00:59:35 on dist-test-slave-7v10"
server_key: "2122e6c7919e422aed92d9ddd2112918"
server_key_iv: "be5a7ddaaedbe06ababb7faa8c100f2b"
server_key_version: "encryptionkey@0"
I20260926 00:59:35.683072 4273 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 00:59:35.683743 4288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:35.683918 4273 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 00:59:35.684021 4273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/ts-2/wal
uuid: "b340ff8635c54e6eb4ed36a2a8613563"
format_stamp: "Formatted at 2026-09-26 00:59:35 on dist-test-slave-7v10"
server_key: "2122e6c7919e422aed92d9ddd2112918"
server_key_iv: "be5a7ddaaedbe06ababb7faa8c100f2b"
server_key_version: "encryptionkey@0"
I20260926 00:59:35.684099 4273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 00:59:35.707388 4273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 00:59:35.710240 4273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 00:59:35.710476 4273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 00:59:35.710765 4273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 00:59:35.711174 4273 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 00:59:35.711241 4273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:35.711295 4273 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 00:59:35.711334 4273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:35.718632 4273 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.3:36443
I20260926 00:59:35.718681 4401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.3:36443 every 8 connection(s)
I20260926 00:59:35.719111 4273 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
Sep 26 00:59:35 dist-test-slave-7v10 krb5kdc[3897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384375, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.3@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 00:59:35.728255 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 4273
I20260926 00:59:35.728368 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384372073861-3372-0/minicluster-data/ts-2/wal/instance
I20260926 00:59:35.728535 3991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:35.719992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.3:60543 (local address 127.3.75.62:38659)
0926 00:59:35.720087 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:35.720090 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:35.720313 (+ 223us) server_negotiation.cc:407] Connection header received
0926 00:59:35.721303 (+ 990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:35.721304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:35.721315 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:35.721381 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:35.721985 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:35.722462 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:35.723521 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:35.723709 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:35.723763 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:35.725694 (+ 1931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:35.725695 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:35.725696 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 00:59:35.725704 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:35.727168 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:35.727575 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:35.727576 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:35.727577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:35.727644 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:35.727905 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:35.727906 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:35.727906 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:35.728019 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:35.728094 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:35.728365 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 00:59:35.728443 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 00:59:35.728544 3372 external_mini_cluster.cc:1428] Setting key 0b08ccedbbb46800c7b8f3f7f83b0332
I20260926 00:59:35.728734 4404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:35.719970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:38659 (local address 127.3.75.3:60543)
0926 00:59:35.720180 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:35.720216 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:35.721164 (+ 948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:35.721440 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:35.721444 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:35.721716 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0926 00:59:35.721907 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:35.721911 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:35.722533 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:35.722533 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:35.723450 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:35.723453 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:35.723786 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:35.723786 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:35.723879 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:35.724312 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 00:59:35.724320 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 00:59:35.725599 (+ 1279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 00:59:35.727265 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 00:59:35.727272 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 00:59:35.727274 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 00:59:35.727497 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 00:59:35.727713 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 00:59:35.727714 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 00:59:35.727715 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 00:59:35.727827 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 00:59:35.728091 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 00:59:35.728098 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 00:59:35.728235 (+ 137us) client_negotiation.cc:769] Sending connection context
0926 00:59:35.728342 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 00:59:35.728500 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":135,"threads_started":1}
I20260926 00:59:35.729133 4402 heartbeater.cc:347] Connected to a master server at 127.3.75.62:38659
I20260926 00:59:35.729249 4402 heartbeater.cc:464] Registering TS with master...
I20260926 00:59:35.729418 4402 heartbeater.cc:511] Master 127.3.75.62:38659 requested a full tablet report, sending...
I20260926 00:59:35.729912 3941 ts_manager.cc:194] Registered new tserver with Master: b340ff8635c54e6eb4ed36a2a8613563 (127.3.75.3:36443)
I20260926 00:59:35.730926 3941 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.75.3@KRBTEST.COM'} at 127.3.75.3:60543
I20260926 00:59:35.740918 3372 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 00:59:35.748554 3991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:35.742383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40324 (local address 127.3.75.62:38659)
0926 00:59:35.742444 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:35.742446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:35.742547 (+ 101us) server_negotiation.cc:407] Connection header received
0926 00:59:35.742617 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:35.742618 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:35.742628 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:35.742699 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:35.743337 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:35.743812 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:35.744534 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:35.744728 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:35.744804 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:35.745977 (+ 1173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:35.745978 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:35.745979 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 00:59:35.745986 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:35.747431 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:35.747777 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:35.747779 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:35.747779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:35.747837 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:35.748031 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:35.748032 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:35.748032 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:35.748175 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:35.748275 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:35.748381 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 00:59:35.748455 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 00:59:35.750723 3941 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40324:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 00:59:35.752321 3941 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 00:59:35.759084 4416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:35.754734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40422 (local address 127.3.75.3:36443)
0926 00:59:35.755312 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:35.755316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:35.755327 (+ 11us) server_negotiation.cc:407] Connection header received
0926 00:59:35.755360 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:35.755362 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:35.755377 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:35.755655 (+ 278us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 00:59:35.755733 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:35.756745 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:35.757844 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:35.758621 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:35.758705 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:35.758771 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:35.758956 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 00:59:35.758998 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":88,"threads_started":1}
I20260926 00:59:35.759130 4415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:35.754635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.3:36443 (local address 127.0.0.1:40422)
0926 00:59:35.754966 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:35.754978 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:35.755038 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:35.755469 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:35.755470 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:35.755471 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 00:59:35.755579 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:35.755581 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:35.756885 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:35.756886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:35.757755 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:35.757758 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:35.758708 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:35.758708 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:35.758812 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:35.758827 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 00:59:35.758934 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 00:59:35.759010 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":29,"spinlock_wait_cycles":768,"thread_start_us":90,"threads_started":1}
I20260926 00:59:35.759371 4418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:35.754916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51428 (local address 127.3.75.1:34373)
0926 00:59:35.755225 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:35.755229 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:35.755239 (+ 10us) server_negotiation.cc:407] Connection header received
0926 00:59:35.755282 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:35.755284 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:35.755300 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:35.755378 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 00:59:35.755625 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:35.756710 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:35.757995 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:35.758685 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:35.758747 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:35.758793 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:35.759163 (+ 370us) server_negotiation.cc:299] Negotiation successful
0926 00:59:35.759314 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":95,"threads_started":1}
I20260926 00:59:35.759130 4413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:35.754635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.1:34373 (local address 127.0.0.1:51428)
0926 00:59:35.754798 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:35.754837 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:35.754935 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:35.755466 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:35.755467 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:35.755469 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 00:59:35.755527 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:35.755530 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:35.756824 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:35.756825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:35.757913 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:35.757915 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:35.758757 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:35.758758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:35.758820 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:35.758830 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 00:59:35.758940 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 00:59:35.759010 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":2,"thread_start_us":61,"threads_started":1}
I20260926 00:59:35.759984 4414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:35.754676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.2:38229 (local address 127.0.0.1:54132)
0926 00:59:35.755127 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:35.755136 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:35.755189 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:35.755384 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:35.755387 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:35.755389 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 00:59:35.755460 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:35.755462 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:35.756483 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:35.756484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:35.757922 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:35.757924 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:35.759720 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:35.759721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:35.759789 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:35.759794 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 00:59:35.759846 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 00:59:35.759906 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"mutex_wait_us":1,"thread_start_us":85,"threads_started":1}
I20260926 00:59:35.760241 4417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:35.754925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54132 (local address 127.3.75.2:38229)
0926 00:59:35.755127 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:35.755132 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:35.755193 (+ 61us) server_negotiation.cc:407] Connection header received
0926 00:59:35.755253 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:35.755255 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:35.755268 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:35.755355 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 00:59:35.755545 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:35.756379 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:35.758828 (+ 2449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:35.759645 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:35.759720 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:35.759784 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:35.760121 (+ 337us) server_negotiation.cc:299] Negotiation successful
0926 00:59:35.760177 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":84,"threads_started":1}
I20260926 00:59:35.760119 4068 tablet_service.cc:1467] Processing CreateTablet for tablet 756b614ce8564653aa2d80645e321074 (DEFAULT_TABLE table=test-table [id=67577a6fcf764cc7b0ac4c14b1295082]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 00:59:35.760119 4336 tablet_service.cc:1467] Processing CreateTablet for tablet 756b614ce8564653aa2d80645e321074 (DEFAULT_TABLE table=test-table [id=67577a6fcf764cc7b0ac4c14b1295082]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 00:59:35.760478 4336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 756b614ce8564653aa2d80645e321074. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 00:59:35.760478 4068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 756b614ce8564653aa2d80645e321074. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 00:59:35.761013 4202 tablet_service.cc:1467] Processing CreateTablet for tablet 756b614ce8564653aa2d80645e321074 (DEFAULT_TABLE table=test-table [id=67577a6fcf764cc7b0ac4c14b1295082]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 00:59:35.761368 4202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 756b614ce8564653aa2d80645e321074. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 00:59:35.763890 4419 tablet_bootstrap.cc:492] T 756b614ce8564653aa2d80645e321074 P b340ff8635c54e6eb4ed36a2a8613563: Bootstrap starting.
I20260926 00:59:35.764110 4420 tablet_bootstrap.cc:492] T 756b614ce8564653aa2d80645e321074 P ad79ef498e8e4076ac30220e9228c3cf: Bootstrap starting.
I20260926 00:59:35.764902 4419 tablet_bootstrap.cc:654] T 756b614ce8564653aa2d80645e321074 P b340ff8635c54e6eb4ed36a2a8613563: Neither blocks nor log segments found. Creating new log.
I20260926 00:59:35.765332 4421 tablet_bootstrap.cc:492] T 756b614ce8564653aa2d80645e321074 P a8b5b8b0bc2046b182164e8b521c28a8: Bootstrap starting.
I20260926 00:59:35.765497 4419 log.cc:824] T 756b614ce8564653aa2d80645e321074 P b340ff8635c54e6eb4ed36a2a8613563: Log is configured to *not* fsync() on all Append() calls
I20260926 00:59:35.766028 4421 tablet_bootstrap.cc:654] T 756b614ce8564653aa2d80645e321074 P a8b5b8b0bc2046b182164e8b521c28a8: Neither blocks nor log segments found. Creating new log.
I20260926 00:59:35.766176 4420 tablet_bootstrap.cc:654] T 756b614ce8564653aa2d80645e321074 P ad79ef498e8e4076ac30220e9228c3cf: Neither blocks nor log segments found. Creating new log.
I20260926 00:59:35.766368 4419 tablet_bootstrap.cc:492] T 756b614ce8564653aa2d80645e321074 P b340ff8635c54e6eb4ed36a2a8613563: No bootstrap required, opened a new log
I20260926 00:59:35.766451 4419 ts_tablet_manager.cc:1397] T 756b614ce8564653aa2d80645e321074 P b340ff8635c54e6eb4ed36a2a8613563: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 00:59:35.766609 4421 log.cc:824] T 756b614ce8564653aa2d80645e321074 P a8b5b8b0bc2046b182164e8b521c28a8: Log is configured to *not* fsync() on all Append() calls
I20260926 00:59:35.766749 4420 log.cc:824] T 756b614ce8564653aa2d80645e321074 P ad79ef498e8e4076ac30220e9228c3cf: Log is configured to *not* fsync() on all Append() calls
I20260926 00:59:35.767443 4420 tablet_bootstrap.cc:492] T 756b614ce8564653aa2d80645e321074 P ad79ef498e8e4076ac30220e9228c3cf: No bootstrap required, opened a new log
I20260926 00:59:35.767499 4421 tablet_bootstrap.cc:492] T 756b614ce8564653aa2d80645e321074 P a8b5b8b0bc2046b182164e8b521c28a8: No bootstrap required, opened a new log
I20260926 00:59:35.767529 4420 ts_tablet_manager.cc:1397] T 756b614ce8564653aa2d80645e321074 P ad79ef498e8e4076ac30220e9228c3cf: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 00:59:35.767568 4421 ts_tablet_manager.cc:1397] T 756b614ce8564653aa2d80645e321074 P a8b5b8b0bc2046b182164e8b521c28a8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 00:59:35.768579 4419 raft_consensus.cc:348] T 756b614ce8564653aa2d80645e321074 P b340ff8635c54e6eb4ed36a2a8613563 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b340ff8635c54e6eb4ed36a2a8613563" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 36443 } } peers { permanent_uuid: "ad79ef498e8e4076ac30220e9228c3cf" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 34373 } } peers { permanent_uuid: "a8b5b8b0bc2046b182164e8b521c28a8" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 38229 } }
I20260926 00:59:35.768762 4419 raft_consensus.cc:374] T 756b614ce8564653aa2d80645e321074 P b340ff8635c54e6eb4ed36a2a8613563 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 00:59:35.768805 4419 raft_consensus.cc:729] T 756b614ce8564653aa2d80645e321074 P b340ff8635c54e6eb4ed36a2a8613563 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b340ff8635c54e6eb4ed36a2a8613563, State: Initialized, Role: FOLLOWER
I20260926 00:59:35.768916 4419 consensus_queue.cc:260] T 756b614ce8564653aa2d80645e321074 P b340ff8635c54e6eb4ed36a2a8613563 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b340ff8635c54e6eb4ed36a2a8613563" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 36443 } } peers { permanent_uuid: "ad79ef498e8e4076ac30220e9228c3cf" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 34373 } } peers { permanent_uuid: "a8b5b8b0bc2046b182164e8b521c28a8" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 38229 } }
I20260926 00:59:35.768913 4420 raft_consensus.cc:348] T 756b614ce8564653aa2d80645e321074 P ad79ef498e8e4076ac30220e9228c3cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b340ff8635c54e6eb4ed36a2a8613563" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 36443 } } peers { permanent_uuid: "ad79ef498e8e4076ac30220e9228c3cf" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 34373 } } peers { permanent_uuid: "a8b5b8b0bc2046b182164e8b521c28a8" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 38229 } }
I20260926 00:59:35.769083 4420 raft_consensus.cc:374] T 756b614ce8564653aa2d80645e321074 P ad79ef498e8e4076ac30220e9228c3cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 00:59:35.769135 4420 raft_consensus.cc:729] T 756b614ce8564653aa2d80645e321074 P ad79ef498e8e4076ac30220e9228c3cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad79ef498e8e4076ac30220e9228c3cf, State: Initialized, Role: FOLLOWER
I20260926 00:59:35.769181 4419 ts_tablet_manager.cc:1428] T 756b614ce8564653aa2d80645e321074 P b340ff8635c54e6eb4ed36a2a8613563: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 00:59:35.769308 4402 heartbeater.cc:503] Master 127.3.75.62:38659 was elected leader, sending a full tablet report...
I20260926 00:59:35.769276 4420 consensus_queue.cc:260] T 756b614ce8564653aa2d80645e321074 P ad79ef498e8e4076ac30220e9228c3cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b340ff8635c54e6eb4ed36a2a8613563" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 36443 } } peers { permanent_uuid: "ad79ef498e8e4076ac30220e9228c3cf" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 34373 } } peers { permanent_uuid: "a8b5b8b0bc2046b182164e8b521c28a8" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 38229 } }
I20260926 00:59:35.769649 4420 ts_tablet_manager.cc:1428] T 756b614ce8564653aa2d80645e321074 P ad79ef498e8e4076ac30220e9228c3cf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 00:59:35.769639 4421 raft_consensus.cc:348] T 756b614ce8564653aa2d80645e321074 P a8b5b8b0bc2046b182164e8b521c28a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b340ff8635c54e6eb4ed36a2a8613563" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 36443 } } peers { permanent_uuid: "ad79ef498e8e4076ac30220e9228c3cf" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 34373 } } peers { permanent_uuid: "a8b5b8b0bc2046b182164e8b521c28a8" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 38229 } }
I20260926 00:59:35.769798 4134 heartbeater.cc:503] Master 127.3.75.62:38659 was elected leader, sending a full tablet report...
I20260926 00:59:35.769809 4421 raft_consensus.cc:374] T 756b614ce8564653aa2d80645e321074 P a8b5b8b0bc2046b182164e8b521c28a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 00:59:35.769845 4421 raft_consensus.cc:729] T 756b614ce8564653aa2d80645e321074 P a8b5b8b0bc2046b182164e8b521c28a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8b5b8b0bc2046b182164e8b521c28a8, State: Initialized, Role: FOLLOWER
I20260926 00:59:35.769956 4421 consensus_queue.cc:260] T 756b614ce8564653aa2d80645e321074 P a8b5b8b0bc2046b182164e8b521c28a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b340ff8635c54e6eb4ed36a2a8613563" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 36443 } } peers { permanent_uuid: "ad79ef498e8e4076ac30220e9228c3cf" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 34373 } } peers { permanent_uuid: "a8b5b8b0bc2046b182164e8b521c28a8" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 38229 } }
I20260926 00:59:35.770231 4421 ts_tablet_manager.cc:1428] T 756b614ce8564653aa2d80645e321074 P a8b5b8b0bc2046b182164e8b521c28a8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 00:59:35.770648 4268 heartbeater.cc:503] Master 127.3.75.62:38659 was elected leader, sending a full tablet report...
W20260926 00:59:35.795512 4269 tablet.cc:2369] T 756b614ce8564653aa2d80645e321074 P a8b5b8b0bc2046b182164e8b521c28a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 00:59:35.882301 4135 tablet.cc:2369] T 756b614ce8564653aa2d80645e321074 P ad79ef498e8e4076ac30220e9228c3cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 00:59:35.969820 4403 tablet.cc:2369] T 756b614ce8564653aa2d80645e321074 P b340ff8635c54e6eb4ed36a2a8613563: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 00:59:35.999248 4425 raft_consensus.cc:482] T 756b614ce8564653aa2d80645e321074 P b340ff8635c54e6eb4ed36a2a8613563 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 00:59:35.999369 4425 raft_consensus.cc:504] T 756b614ce8564653aa2d80645e321074 P b340ff8635c54e6eb4ed36a2a8613563 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b340ff8635c54e6eb4ed36a2a8613563" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 36443 } } peers { permanent_uuid: "ad79ef498e8e4076ac30220e9228c3cf" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 34373 } } peers { permanent_uuid: "a8b5b8b0bc2046b182164e8b521c28a8" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 38229 } }
I20260926 00:59:35.999817 4425 leader_election.cc:288] T 756b614ce8564653aa2d80645e321074 P b340ff8635c54e6eb4ed36a2a8613563 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad79ef498e8e4076ac30220e9228c3cf (127.3.75.1:34373), a8b5b8b0bc2046b182164e8b521c28a8 (127.3.75.2:38229)
I20260926 00:59:36.003237 4418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:36.000170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.3:50363 (local address 127.3.75.1:34373)
0926 00:59:36.000242 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:36.000244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:36.000496 (+ 252us) server_negotiation.cc:407] Connection header received
0926 00:59:36.000568 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:36.000569 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:36.000580 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:36.000620 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 00:59:36.000777 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:36.001259 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:36.002240 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:36.002793 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:36.002894 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:36.002905 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:36.003116 (+ 211us) server_negotiation.cc:299] Negotiation successful
0926 00:59:36.003161 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 00:59:36.003325 4404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:36.000043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.2:38229 (local address 127.3.75.3:46183)
0926 00:59:36.000114 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:36.000128 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:36.000190 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:36.000326 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:36.000327 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:36.000328 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 00:59:36.000400 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:36.000402 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:36.001050 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:36.001051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:36.002272 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:36.002275 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:36.003058 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:36.003059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:36.003141 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:36.003144 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 00:59:36.003194 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 00:59:36.003229 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 00:59:36.003413 4428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:36.000050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.1:34373 (local address 127.3.75.3:50363)
0926 00:59:36.000411 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:36.000420 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:36.000498 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:36.000631 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:36.000632 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:36.000634 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 00:59:36.000705 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:36.000707 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:36.001364 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:36.001365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:36.002140 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:36.002142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:36.002910 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:36.002911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:36.002994 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:36.002999 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 00:59:36.003101 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 00:59:36.003152 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":168,"thread_start_us":126,"threads_started":1}
I20260926 00:59:36.003734 4088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "756b614ce8564653aa2d80645e321074" candidate_uuid: "b340ff8635c54e6eb4ed36a2a8613563" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad79ef498e8e4076ac30220e9228c3cf" is_pre_election: true
I20260926 00:59:36.003890 4088 raft_consensus.cc:2454] T 756b614ce8564653aa2d80645e321074 P ad79ef498e8e4076ac30220e9228c3cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b340ff8635c54e6eb4ed36a2a8613563 in term 0.
I20260926 00:59:36.003569 4417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:36.000152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.3:46183 (local address 127.3.75.2:38229)
0926 00:59:36.000211 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:36.000213 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:36.000222 (+ 9us) server_negotiation.cc:407] Connection header received
0926 00:59:36.000250 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:36.000251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:36.000262 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:36.000312 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 00:59:36.000474 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:36.000961 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:36.002382 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:36.002974 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:36.003057 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:36.003067 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:36.003278 (+ 211us) server_negotiation.cc:299] Negotiation successful
0926 00:59:36.003481 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 00:59:36.004148 4291 leader_election.cc:302] T 756b614ce8564653aa2d80645e321074 P b340ff8635c54e6eb4ed36a2a8613563 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad79ef498e8e4076ac30220e9228c3cf, b340ff8635c54e6eb4ed36a2a8613563; no voters:
I20260926 00:59:36.004287 4425 raft_consensus.cc:2781] T 756b614ce8564653aa2d80645e321074 P b340ff8635c54e6eb4ed36a2a8613563 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 00:59:36.004334 4425 raft_consensus.cc:482] T 756b614ce8564653aa2d80645e321074 P b340ff8635c54e6eb4ed36a2a8613563 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 00:59:36.004364 4425 raft_consensus.cc:3037] T 756b614ce8564653aa2d80645e321074 P b340ff8635c54e6eb4ed36a2a8613563 [term 0 FOLLOWER]: Advancing to term 1
I20260926 00:59:36.004285 4222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "756b614ce8564653aa2d80645e321074" candidate_uuid: "b340ff8635c54e6eb4ed36a2a8613563" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8b5b8b0bc2046b182164e8b521c28a8" is_pre_election: true
I20260926 00:59:36.004449 4222 raft_consensus.cc:2454] T 756b614ce8564653aa2d80645e321074 P a8b5b8b0bc2046b182164e8b521c28a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b340ff8635c54e6eb4ed36a2a8613563 in term 0.
I20260926 00:59:36.005406 4425 raft_consensus.cc:504] T 756b614ce8564653aa2d80645e321074 P b340ff8635c54e6eb4ed36a2a8613563 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b340ff8635c54e6eb4ed36a2a8613563" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 36443 } } peers { permanent_uuid: "ad79ef498e8e4076ac30220e9228c3cf" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 34373 } } peers { permanent_uuid: "a8b5b8b0bc2046b182164e8b521c28a8" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 38229 } }
I20260926 00:59:36.005620 4425 leader_election.cc:288] T 756b614ce8564653aa2d80645e321074 P b340ff8635c54e6eb4ed36a2a8613563 [CANDIDATE]: Term 1 election: Requested vote from peers ad79ef498e8e4076ac30220e9228c3cf (127.3.75.1:34373), a8b5b8b0bc2046b182164e8b521c28a8 (127.3.75.2:38229)
I20260926 00:59:36.005697 4088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "756b614ce8564653aa2d80645e321074" candidate_uuid: "b340ff8635c54e6eb4ed36a2a8613563" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad79ef498e8e4076ac30220e9228c3cf"
I20260926 00:59:36.005716 4222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "756b614ce8564653aa2d80645e321074" candidate_uuid: "b340ff8635c54e6eb4ed36a2a8613563" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8b5b8b0bc2046b182164e8b521c28a8"
I20260926 00:59:36.005827 4222 raft_consensus.cc:3037] T 756b614ce8564653aa2d80645e321074 P a8b5b8b0bc2046b182164e8b521c28a8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 00:59:36.005829 4088 raft_consensus.cc:3037] T 756b614ce8564653aa2d80645e321074 P ad79ef498e8e4076ac30220e9228c3cf [term 0 FOLLOWER]: Advancing to term 1
I20260926 00:59:36.101750 4222 raft_consensus.cc:2454] T 756b614ce8564653aa2d80645e321074 P a8b5b8b0bc2046b182164e8b521c28a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b340ff8635c54e6eb4ed36a2a8613563 in term 1.
I20260926 00:59:36.101785 4088 raft_consensus.cc:2454] T 756b614ce8564653aa2d80645e321074 P ad79ef498e8e4076ac30220e9228c3cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b340ff8635c54e6eb4ed36a2a8613563 in term 1.
I20260926 00:59:36.102053 4289 leader_election.cc:302] T 756b614ce8564653aa2d80645e321074 P b340ff8635c54e6eb4ed36a2a8613563 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8b5b8b0bc2046b182164e8b521c28a8, b340ff8635c54e6eb4ed36a2a8613563; no voters:
I20260926 00:59:36.102203 4425 raft_consensus.cc:2781] T 756b614ce8564653aa2d80645e321074 P b340ff8635c54e6eb4ed36a2a8613563 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 00:59:36.102368 4425 raft_consensus.cc:686] T 756b614ce8564653aa2d80645e321074 P b340ff8635c54e6eb4ed36a2a8613563 [term 1 LEADER]: Becoming Leader. State: Replica: b340ff8635c54e6eb4ed36a2a8613563, State: Running, Role: LEADER
I20260926 00:59:36.102473 4425 consensus_queue.cc:237] T 756b614ce8564653aa2d80645e321074 P b340ff8635c54e6eb4ed36a2a8613563 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b340ff8635c54e6eb4ed36a2a8613563" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 36443 } } peers { permanent_uuid: "ad79ef498e8e4076ac30220e9228c3cf" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 34373 } } peers { permanent_uuid: "a8b5b8b0bc2046b182164e8b521c28a8" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 38229 } }
I20260926 00:59:36.103386 3942 catalog_manager.cc:5495] T 756b614ce8564653aa2d80645e321074 P b340ff8635c54e6eb4ed36a2a8613563 reported cstate change: term changed from 0 to 1, leader changed from <none> to b340ff8635c54e6eb4ed36a2a8613563 (127.3.75.3). New cstate: current_term: 1 leader_uuid: "b340ff8635c54e6eb4ed36a2a8613563" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b340ff8635c54e6eb4ed36a2a8613563" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 36443 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad79ef498e8e4076ac30220e9228c3cf" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 34373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8b5b8b0bc2046b182164e8b521c28a8" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 38229 } health_report { overall_health: UNKNOWN } } }
Sep 26 00:59:36 dist-test-slave-7v10 krb5kdc[3897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.75.1@KRBTEST.COM
I20260926 00:59:36.120615 4418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:36.112825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51444 (local address 127.3.75.1:34373)
0926 00:59:36.112900 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:36.112902 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:36.113017 (+ 115us) server_negotiation.cc:407] Connection header received
0926 00:59:36.113140 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:36.113141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:36.113150 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:36.113199 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:36.114046 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:36.114551 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:36.115393 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:36.115575 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:36.115636 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:36.117934 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:36.117946 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:36.117946 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 00:59:36.117953 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:36.119635 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:36.119933 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:36.119936 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:36.119936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:36.119996 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:36.120167 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:36.120168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:36.120168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:36.120291 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:36.120385 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:36.120444 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 00:59:36.120504 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Sep 26 00:59:36 dist-test-slave-7v10 krb5kdc[3897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384376, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 00:59:36 dist-test-slave-7v10 krb5kdc[3897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384376, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.75.1@KRBTEST.COM
I20260926 00:59:36.208395 4418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:36.200635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51460 (local address 127.3.75.1:34373)
0926 00:59:36.200710 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:36.200712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:36.200846 (+ 134us) server_negotiation.cc:407] Connection header received
0926 00:59:36.200917 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:36.200918 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:36.200927 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:36.200979 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:36.201771 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:36.202260 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:36.203480 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:36.203669 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:36.203726 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:36.206100 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:36.206101 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:36.206101 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 00:59:36.206108 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:36.207392 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:36.207705 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:36.207706 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:36.207706 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:36.207754 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:36.207911 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:36.207912 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:36.207912 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:36.207995 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:36.208062 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:36.208216 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 00:59:36.208279 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Sep 26 00:59:36 dist-test-slave-7v10 krb5kdc[3897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 00:59:36 dist-test-slave-7v10 krb5kdc[3897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.75.1@KRBTEST.COM
I20260926 00:59:36.306751 4418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:36.299767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51468 (local address 127.3.75.1:34373)
0926 00:59:36.299821 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:36.299823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:36.299912 (+ 89us) server_negotiation.cc:407] Connection header received
0926 00:59:36.300013 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:36.300014 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:36.300024 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:36.300094 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:36.300659 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:36.301136 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:36.301898 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:36.302095 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:36.302180 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:36.304441 (+ 2261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:36.304443 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:36.304443 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 00:59:36.304452 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:36.305601 (+ 1149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:36.305958 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:36.305959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:36.305960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:36.306022 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:36.306246 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:36.306247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:36.306247 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:36.306348 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:36.306453 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:36.306562 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 00:59:36.306645 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
W20260926 00:59:36.307215 4048 tablet_service.cc:2908] Rejecting scan request for tablet 756b614ce8564653aa2d80645e321074: Uninitialized: clean time has not yet been initialized
I20260926 00:59:36.308231 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 4005
I20260926 00:59:36.313048 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 4139
I20260926 00:59:36.318378 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 4273
I20260926 00:59:36.323027 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 3910
2026-09-26T00:59:36Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2406 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 00:59:38 dist-test-slave-7v10 krb5kdc[4443](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 00:59:38 dist-test-slave-7v10 krb5kdc[4443](info): set up 2 sockets
Sep 26 00:59:38 dist-test-slave-7v10 krb5kdc[4443](info): commencing operation
krb5kdc: starting...
W20260926 00:59:39.204957 3372 mini_kdc.cc:121] Time spent starting KDC: real 2.861s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260926 00:59:39.316440 3372 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.111s user 0.000s sys 0.001s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/krb5kdc/test-user.keytab.
W20260926 00:59:40.109687 3372 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.632s user 0.000s sys 0.001s
Sep 26 00:59:40 dist-test-slave-7v10 krb5kdc[4443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T00:59:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T00:59:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.75.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab.
Entry for principal kudu/127.3.75.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab.
W20260926 00:59:41.104938 3372 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.3.75.62: real 0.974s user 0.002s sys 0.000s
I20260926 00:59:41.105274 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:34681
--webserver_interface=127.3.75.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:37913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.75.62:34681
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 00:59:41.187012 4456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 00:59:41.187201 4456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 00:59:41.187222 4456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 00:59:41.188234 4456 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 00:59:41.188267 4456 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 00:59:41.188280 4456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 00:59:41.188293 4456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 00:59:41.188305 4456 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 00:59:41.188316 4456 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 00:59:41.189787 4456 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:37913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.75.62:34681
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:34681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 00:59:41.189971 4456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 00:59:41.190125 4456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 00:59:41.194856 4461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:41.194932 4464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 00:59:41.195052 4456 server_base.cc:1034] running on GCE node
W20260926 00:59:41.194865 4462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 00:59:41.195415 4456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 00:59:41.195732 4456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 00:59:41.196902 4456 hybrid_clock.cc:648] HybridClock initialized: now 1790384381196878 us; error 36 us; skew 500 ppm
Sep 26 00:59:41 dist-test-slave-7v10 krb5kdc[4443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384381, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 00:59:41.203488 4456 init.cc:377] Logged in from keytab as kudu/127.3.75.62@KRBTEST.COM (short username kudu)
I20260926 00:59:41.203858 4456 webserver.cc:460] Webserver started at http://127.3.75.62:35019/ using document root <none> and password file <none>
I20260926 00:59:41.204042 4456 fs_manager.cc:359] Metadata directory not provided
I20260926 00:59:41.204082 4456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 00:59:41.204166 4456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 00:59:41.205139 4456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/master-0/data/instance:
uuid: "03a7c52ba87c4b0a875dec905ad499a1"
format_stamp: "Formatted at 2026-09-26 00:59:41 on dist-test-slave-7v10"
server_key: "eda142afc3cbd71c341b6b5b9443a039"
server_key_iv: "b1d703057ce1a6ad29c78bcc21d89d23"
server_key_version: "encryptionkey@0"
I20260926 00:59:41.205430 4456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/master-0/wal/instance:
uuid: "03a7c52ba87c4b0a875dec905ad499a1"
format_stamp: "Formatted at 2026-09-26 00:59:41 on dist-test-slave-7v10"
server_key: "eda142afc3cbd71c341b6b5b9443a039"
server_key_iv: "b1d703057ce1a6ad29c78bcc21d89d23"
server_key_version: "encryptionkey@0"
I20260926 00:59:41.206681 4456 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 00:59:41.207418 4471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:41.207590 4456 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:41.207684 4456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/master-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/master-0/wal
uuid: "03a7c52ba87c4b0a875dec905ad499a1"
format_stamp: "Formatted at 2026-09-26 00:59:41 on dist-test-slave-7v10"
server_key: "eda142afc3cbd71c341b6b5b9443a039"
server_key_iv: "b1d703057ce1a6ad29c78bcc21d89d23"
server_key_version: "encryptionkey@0"
I20260926 00:59:41.207737 4456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 00:59:41.217023 4456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 00:59:41.219316 4456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 00:59:41.219427 4456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 00:59:41.223825 4456 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.62:34681
I20260926 00:59:41.223913 4533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.62:34681 every 8 connection(s)
I20260926 00:59:41.224241 4456 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
I20260926 00:59:41.224941 4534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 00:59:41.227284 4534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03a7c52ba87c4b0a875dec905ad499a1: Bootstrap starting.
I20260926 00:59:41.227875 4534 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03a7c52ba87c4b0a875dec905ad499a1: Neither blocks nor log segments found. Creating new log.
I20260926 00:59:41.228173 4534 log.cc:824] T 00000000000000000000000000000000 P 03a7c52ba87c4b0a875dec905ad499a1: Log is configured to *not* fsync() on all Append() calls
I20260926 00:59:41.228855 4534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03a7c52ba87c4b0a875dec905ad499a1: No bootstrap required, opened a new log
I20260926 00:59:41.230121 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 4456
I20260926 00:59:41.230204 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/master-0/wal/instance
I20260926 00:59:41.230361 3372 external_mini_cluster.cc:1428] Setting key c78b6885e9e1fd361e314171be698a13
I20260926 00:59:41.231382 4534 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03a7c52ba87c4b0a875dec905ad499a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03a7c52ba87c4b0a875dec905ad499a1" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 34681 } }
I20260926 00:59:41.231563 4534 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03a7c52ba87c4b0a875dec905ad499a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 00:59:41.231603 4534 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03a7c52ba87c4b0a875dec905ad499a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03a7c52ba87c4b0a875dec905ad499a1, State: Initialized, Role: FOLLOWER
I20260926 00:59:41.231796 4534 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03a7c52ba87c4b0a875dec905ad499a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03a7c52ba87c4b0a875dec905ad499a1" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 34681 } }
I20260926 00:59:41.231892 4534 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03a7c52ba87c4b0a875dec905ad499a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 00:59:41.231926 4534 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03a7c52ba87c4b0a875dec905ad499a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 00:59:41.231982 4534 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03a7c52ba87c4b0a875dec905ad499a1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 00:59:41.232643 4534 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03a7c52ba87c4b0a875dec905ad499a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03a7c52ba87c4b0a875dec905ad499a1" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 34681 } }
I20260926 00:59:41.232728 4534 leader_election.cc:302] T 00000000000000000000000000000000 P 03a7c52ba87c4b0a875dec905ad499a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03a7c52ba87c4b0a875dec905ad499a1; no voters:
I20260926 00:59:41.232936 4534 leader_election.cc:288] T 00000000000000000000000000000000 P 03a7c52ba87c4b0a875dec905ad499a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 00:59:41.232986 4539 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03a7c52ba87c4b0a875dec905ad499a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 00:59:41.233151 4539 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03a7c52ba87c4b0a875dec905ad499a1 [term 1 LEADER]: Becoming Leader. State: Replica: 03a7c52ba87c4b0a875dec905ad499a1, State: Running, Role: LEADER
I20260926 00:59:41.233201 4534 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03a7c52ba87c4b0a875dec905ad499a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 00:59:41.233258 4539 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03a7c52ba87c4b0a875dec905ad499a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03a7c52ba87c4b0a875dec905ad499a1" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 34681 } }
I20260926 00:59:41.233650 4541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03a7c52ba87c4b0a875dec905ad499a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03a7c52ba87c4b0a875dec905ad499a1. Latest consensus state: current_term: 1 leader_uuid: "03a7c52ba87c4b0a875dec905ad499a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03a7c52ba87c4b0a875dec905ad499a1" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 34681 } } }
I20260926 00:59:41.233639 4540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03a7c52ba87c4b0a875dec905ad499a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03a7c52ba87c4b0a875dec905ad499a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03a7c52ba87c4b0a875dec905ad499a1" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 34681 } } }
I20260926 00:59:41.233742 4541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03a7c52ba87c4b0a875dec905ad499a1 [sys.catalog]: This master's current role is: LEADER
I20260926 00:59:41.233748 4540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03a7c52ba87c4b0a875dec905ad499a1 [sys.catalog]: This master's current role is: LEADER
I20260926 00:59:41.234076 4545 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 00:59:41.234604 4545 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 00:59:41.236562 4545 catalog_manager.cc:1347] Generated new cluster ID: 427e7109ef90430f956545533cd67461
I20260926 00:59:41.236613 4545 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 00:59:41 dist-test-slave-7v10 krb5kdc[4443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 00:59:41.242141 4545 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 00:59:41.242622 4538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:41.230947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39498 (local address 127.3.75.62:34681)
0926 00:59:41.231128 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:41.231141 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:41.231166 (+ 25us) server_negotiation.cc:407] Connection header received
0926 00:59:41.231359 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:41.231365 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:41.231424 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:41.231543 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:41.232389 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:41.233734 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:41.234850 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:41.235544 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:41.235651 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:41.238444 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:41.238448 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:41.238448 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 00:59:41.238461 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:41.240580 (+ 2119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:41.240986 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:41.240989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:41.240990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:41.241139 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:41.241355 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:41.241355 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:41.241356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:41.241663 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:41.242206 (+ 543us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:41.242283 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 00:59:41.242423 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":96,"threads_started":1}
I20260926 00:59:41.242625 4545 catalog_manager.cc:1504] Loading token signing keys...
I20260926 00:59:41.248128 4545 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03a7c52ba87c4b0a875dec905ad499a1: Generated new TSK 0
I20260926 00:59:41.248286 4545 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.75.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab.
Entry for principal kudu/127.3.75.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab.
I20260926 00:59:41.277974 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.1:0
--local_ip_for_outbound_sockets=127.3.75.1
--webserver_interface=127.3.75.1
--webserver_port=0
--tserver_master_addrs=127.3.75.62:34681
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab
--principal=kudu/127.3.75.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:37913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 00:59:41.357878 4560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 00:59:41.358078 4560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 00:59:41.358100 4560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 00:59:41.359304 4560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 00:59:41.359345 4560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 00:59:41.359366 4560 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 00:59:41.359385 4560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.1
I20260926 00:59:41.360800 4560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:37913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab
--principal=kudu/127.3.75.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.75.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.75.62:34681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.1
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 00:59:41.361029 4560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 00:59:41.361274 4560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 00:59:41.366251 4565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:41.366240 4566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 00:59:41.366274 4560 server_base.cc:1034] running on GCE node
W20260926 00:59:41.366245 4568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 00:59:41.366572 4560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 00:59:41.366801 4560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 00:59:41.368000 4560 hybrid_clock.cc:648] HybridClock initialized: now 1790384381367986 us; error 29 us; skew 500 ppm
Sep 26 00:59:41 dist-test-slave-7v10 krb5kdc[4443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384381, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 00:59:41.370370 4560 init.cc:377] Logged in from keytab as kudu/127.3.75.1@KRBTEST.COM (short username kudu)
I20260926 00:59:41.370783 4560 webserver.cc:460] Webserver started at http://127.3.75.1:36165/ using document root <none> and password file <none>
I20260926 00:59:41.371021 4560 fs_manager.cc:359] Metadata directory not provided
I20260926 00:59:41.371066 4560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 00:59:41.371208 4560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 00:59:41.372228 4560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/ts-0/data/instance:
uuid: "507869ed7bf44153a968857556049df0"
format_stamp: "Formatted at 2026-09-26 00:59:41 on dist-test-slave-7v10"
server_key: "f74ed9f48e9f758362ac740e48645d69"
server_key_iv: "6310e2e85bfbf8597007696efa42ded0"
server_key_version: "encryptionkey@0"
I20260926 00:59:41.372596 4560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/ts-0/wal/instance:
uuid: "507869ed7bf44153a968857556049df0"
format_stamp: "Formatted at 2026-09-26 00:59:41 on dist-test-slave-7v10"
server_key: "f74ed9f48e9f758362ac740e48645d69"
server_key_iv: "6310e2e85bfbf8597007696efa42ded0"
server_key_version: "encryptionkey@0"
I20260926 00:59:41.373939 4560 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 00:59:41.374624 4575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:41.374773 4560 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 00:59:41.374856 4560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/ts-0/wal
uuid: "507869ed7bf44153a968857556049df0"
format_stamp: "Formatted at 2026-09-26 00:59:41 on dist-test-slave-7v10"
server_key: "f74ed9f48e9f758362ac740e48645d69"
server_key_iv: "6310e2e85bfbf8597007696efa42ded0"
server_key_version: "encryptionkey@0"
I20260926 00:59:41.374955 4560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 00:59:41.387091 4560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 00:59:41.388814 4560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 00:59:41.388919 4560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 00:59:41.389367 4560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 00:59:41.389405 4560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:41.389432 4560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 00:59:41.389446 4560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:41.396093 4560 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.1:37775
I20260926 00:59:41.396165 4690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.1:37775 every 8 connection(s)
I20260926 00:59:41.396440 4560 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
Sep 26 00:59:41 dist-test-slave-7v10 krb5kdc[4443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384381, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.1@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 00:59:41.399659 4538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:41.389891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:54995 (local address 127.3.75.62:34681)
0926 00:59:41.389965 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:41.389968 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:41.390143 (+ 175us) server_negotiation.cc:407] Connection header received
0926 00:59:41.390619 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:41.390620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:41.390639 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:41.390717 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:41.391614 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:41.392519 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:41.394069 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:41.394423 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:41.394509 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:41.397059 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:41.397060 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:41.397060 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 00:59:41.397069 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:41.398483 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:41.398834 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:41.398835 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:41.398835 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:41.398936 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:41.399142 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:41.399142 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:41.399143 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:41.399243 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:41.399293 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:41.399523 (+ 230us) server_negotiation.cc:299] Negotiation successful
0926 00:59:41.399599 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 00:59:41.399809 4584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:41.389821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34681 (local address 127.3.75.1:54995)
0926 00:59:41.390016 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:41.390057 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:41.390521 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:41.390815 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:41.390819 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:41.391221 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0926 00:59:41.391504 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:41.391511 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:41.392619 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:41.392620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:41.393978 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:41.393981 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:41.394521 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:41.394522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:41.394657 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:41.395236 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 00:59:41.395247 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 00:59:41.396970 (+ 1723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 00:59:41.398564 (+ 1594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 00:59:41.398569 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 00:59:41.398571 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 00:59:41.398770 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 00:59:41.398984 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 00:59:41.398989 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 00:59:41.398989 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 00:59:41.399086 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 00:59:41.399297 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 00:59:41.399302 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 00:59:41.399406 (+ 104us) client_negotiation.cc:769] Sending connection context
0926 00:59:41.399485 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 00:59:41.399608 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":92,"threads_started":1}
I20260926 00:59:41.400285 4691 heartbeater.cc:347] Connected to a master server at 127.3.75.62:34681
I20260926 00:59:41.400368 4691 heartbeater.cc:464] Registering TS with master...
I20260926 00:59:41.400542 4691 heartbeater.cc:511] Master 127.3.75.62:34681 requested a full tablet report, sending...
I20260926 00:59:41.401026 4487 ts_manager.cc:194] Registered new tserver with Master: 507869ed7bf44153a968857556049df0 (127.3.75.1:37775)
I20260926 00:59:41.401870 4487 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.75.1@KRBTEST.COM'} at 127.3.75.1:54995
I20260926 00:59:41.402688 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 4560
I20260926 00:59:41.402793 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/ts-0/wal/instance
I20260926 00:59:41.403002 3372 external_mini_cluster.cc:1428] Setting key dd64f3dea4b55fa948865e24624e7743
I20260926 00:59:41.406801 4538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:41.401378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:40819 (local address 127.3.75.62:34681)
0926 00:59:41.401431 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:41.401432 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:41.401500 (+ 68us) server_negotiation.cc:407] Connection header received
0926 00:59:41.401543 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:41.401544 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:41.401553 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:41.401608 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:41.401857 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:41.402445 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:41.403227 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:41.403485 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:41.403540 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:41.404432 (+ 892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:41.404434 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:41.404434 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 00:59:41.404442 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:41.405783 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:41.406165 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:41.406166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:41.406166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:41.406211 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:41.406401 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:41.406402 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:41.406402 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:41.406488 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:41.406545 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:41.406696 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 00:59:41.406749 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 00:59:41.406965 4697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:41.401292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34681 (local address 127.3.75.1:40819)
0926 00:59:41.401433 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:41.401443 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:41.401497 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:41.401611 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:41.401612 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:41.401750 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0926 00:59:41.401800 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:41.401802 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:41.402563 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:41.402564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:41.403132 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:41.403133 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:41.403555 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:41.403556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:41.403631 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:41.404061 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 00:59:41.404065 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 00:59:41.404355 (+ 290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 00:59:41.405862 (+ 1507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 00:59:41.405864 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 00:59:41.405864 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 00:59:41.406094 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 00:59:41.406273 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 00:59:41.406274 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 00:59:41.406275 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 00:59:41.406344 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 00:59:41.406550 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 00:59:41.406551 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 00:59:41.406606 (+ 55us) client_negotiation.cc:769] Sending connection context
0926 00:59:41.406682 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 00:59:41.406792 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":80,"threads_started":1}
WARNING: no policy specified for kudu/127.3.75.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab.
Entry for principal kudu/127.3.75.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab.
I20260926 00:59:41.425773 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.2:0
--local_ip_for_outbound_sockets=127.3.75.2
--webserver_interface=127.3.75.2
--webserver_port=0
--tserver_master_addrs=127.3.75.62:34681
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab
--principal=kudu/127.3.75.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:37913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 00:59:41.504652 4700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 00:59:41.504841 4700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 00:59:41.504864 4700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 00:59:41.505928 4700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 00:59:41.505963 4700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 00:59:41.505985 4700 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 00:59:41.506008 4700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.2
I20260926 00:59:41.507426 4700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:37913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab
--principal=kudu/127.3.75.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.75.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.75.62:34681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.2
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 00:59:41.507601 4700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 00:59:41.507761 4700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 00:59:41.512594 4708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:41.512643 4705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 00:59:41.512768 4700 server_base.cc:1034] running on GCE node
W20260926 00:59:41.512594 4706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 00:59:41.513057 4700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 00:59:41.513302 4700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 00:59:41.514498 4700 hybrid_clock.cc:648] HybridClock initialized: now 1790384381514479 us; error 52 us; skew 500 ppm
Sep 26 00:59:41 dist-test-slave-7v10 krb5kdc[4443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384381, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 00:59:41.517493 4700 init.cc:377] Logged in from keytab as kudu/127.3.75.2@KRBTEST.COM (short username kudu)
I20260926 00:59:41.517969 4700 webserver.cc:460] Webserver started at http://127.3.75.2:39137/ using document root <none> and password file <none>
I20260926 00:59:41.518206 4700 fs_manager.cc:359] Metadata directory not provided
I20260926 00:59:41.518250 4700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 00:59:41.518394 4700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 00:59:41.519516 4700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/ts-1/data/instance:
uuid: "9066d18b104344189422cfe810192780"
format_stamp: "Formatted at 2026-09-26 00:59:41 on dist-test-slave-7v10"
server_key: "2246bbcc48ffdc3a40e2393ddcc69f15"
server_key_iv: "1bd12d6b242c6ff0ebd31bdf8aeeb51b"
server_key_version: "encryptionkey@0"
I20260926 00:59:41.519903 4700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/ts-1/wal/instance:
uuid: "9066d18b104344189422cfe810192780"
format_stamp: "Formatted at 2026-09-26 00:59:41 on dist-test-slave-7v10"
server_key: "2246bbcc48ffdc3a40e2393ddcc69f15"
server_key_iv: "1bd12d6b242c6ff0ebd31bdf8aeeb51b"
server_key_version: "encryptionkey@0"
I20260926 00:59:41.521287 4700 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 00:59:41.521991 4715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:41.522171 4700 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 00:59:41.522262 4700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/ts-1/wal
uuid: "9066d18b104344189422cfe810192780"
format_stamp: "Formatted at 2026-09-26 00:59:41 on dist-test-slave-7v10"
server_key: "2246bbcc48ffdc3a40e2393ddcc69f15"
server_key_iv: "1bd12d6b242c6ff0ebd31bdf8aeeb51b"
server_key_version: "encryptionkey@0"
I20260926 00:59:41.522336 4700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 00:59:41.543872 4700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 00:59:41.546229 4700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 00:59:41.546342 4700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 00:59:41.547011 4700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 00:59:41.547073 4700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:41.547117 4700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 00:59:41.547147 4700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 00:59:41 dist-test-slave-7v10 krb5kdc[4443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384381, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.2@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 00:59:41.554698 4700 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.2:35513
I20260926 00:59:41.554796 4830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.2:35513 every 8 connection(s)
I20260926 00:59:41.555141 4700 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
I20260926 00:59:41.557415 4538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:41.547351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:46489 (local address 127.3.75.62:34681)
0926 00:59:41.547401 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:41.547403 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:41.547651 (+ 248us) server_negotiation.cc:407] Connection header received
0926 00:59:41.548378 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:41.548379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:41.548390 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:41.548432 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:41.549345 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:41.549836 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:41.551064 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:41.551264 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:41.551350 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:41.554784 (+ 3434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:41.554786 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:41.554787 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 00:59:41.554795 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:41.556209 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:41.556585 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:41.556586 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:41.556587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:41.556638 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:41.556864 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:41.556865 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:41.556866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:41.556988 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:41.557062 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:41.557285 (+ 223us) server_negotiation.cc:299] Negotiation successful
0926 00:59:41.557353 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 00:59:41.557552 4724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:41.547350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34681 (local address 127.3.75.2:46489)
0926 00:59:41.547507 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:41.547548 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:41.548271 (+ 723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:41.548527 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:41.548531 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:41.548923 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0926 00:59:41.549245 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:41.549254 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:41.549943 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:41.549945 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:41.550969 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:41.550972 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:41.551361 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:41.551363 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:41.551485 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:41.552042 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 00:59:41.552053 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 00:59:41.554666 (+ 2613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 00:59:41.556281 (+ 1615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 00:59:41.556287 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 00:59:41.556288 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 00:59:41.556505 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 00:59:41.556695 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 00:59:41.556696 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 00:59:41.556696 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 00:59:41.556783 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 00:59:41.557065 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 00:59:41.557069 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 00:59:41.557180 (+ 111us) client_negotiation.cc:769] Sending connection context
0926 00:59:41.557261 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 00:59:41.557378 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":83,"threads_started":1}
I20260926 00:59:41.557989 4831 heartbeater.cc:347] Connected to a master server at 127.3.75.62:34681
I20260926 00:59:41.558080 4831 heartbeater.cc:464] Registering TS with master...
I20260926 00:59:41.558254 4831 heartbeater.cc:511] Master 127.3.75.62:34681 requested a full tablet report, sending...
I20260926 00:59:41.558609 4488 ts_manager.cc:194] Registered new tserver with Master: 9066d18b104344189422cfe810192780 (127.3.75.2:35513)
I20260926 00:59:41.559172 4488 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.75.2@KRBTEST.COM'} at 127.3.75.2:46489
I20260926 00:59:41.560487 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 4700
I20260926 00:59:41.560580 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/ts-1/wal/instance
I20260926 00:59:41.560765 3372 external_mini_cluster.cc:1428] Setting key 086c91e662d5f6106ac81317f6ecb53f
I20260926 00:59:41.564999 4538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:41.559062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:43649 (local address 127.3.75.62:34681)
0926 00:59:41.559102 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:41.559103 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:41.559178 (+ 75us) server_negotiation.cc:407] Connection header received
0926 00:59:41.559280 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:41.559281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:41.559293 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:41.559347 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:41.559699 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:41.560218 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:41.561249 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:41.561498 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:41.561579 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:41.562437 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:41.562439 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:41.562439 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 00:59:41.562447 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:41.563860 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:41.564214 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:41.564215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:41.564216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:41.564269 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:41.564476 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:41.564476 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:41.564477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:41.564614 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:41.564686 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:41.564844 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 00:59:41.564915 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 00:59:41.565016 4837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:41.558986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34681 (local address 127.3.75.2:43649)
0926 00:59:41.559127 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:41.559137 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:41.559193 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:41.559360 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:41.559361 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:41.559565 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0926 00:59:41.559622 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:41.559624 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:41.560324 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:41.560325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:41.561157 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:41.561159 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:41.561578 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:41.561579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:41.561658 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:41.562069 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 00:59:41.562073 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 00:59:41.562333 (+ 260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 00:59:41.563962 (+ 1629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 00:59:41.563963 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 00:59:41.563964 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 00:59:41.564131 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 00:59:41.564338 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 00:59:41.564339 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 00:59:41.564340 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 00:59:41.564395 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 00:59:41.564692 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 00:59:41.564693 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 00:59:41.564734 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 00:59:41.564819 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 00:59:41.564916 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":83,"threads_started":1}
WARNING: no policy specified for kudu/127.3.75.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab.
Entry for principal kudu/127.3.75.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab.
I20260926 00:59:41.584023 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.3:0
--local_ip_for_outbound_sockets=127.3.75.3
--webserver_interface=127.3.75.3
--webserver_port=0
--tserver_master_addrs=127.3.75.62:34681
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab
--principal=kudu/127.3.75.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:37913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 00:59:41.667460 4840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 00:59:41.667704 4840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 00:59:41.667743 4840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 00:59:41.668867 4840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 00:59:41.668932 4840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 00:59:41.668962 4840 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 00:59:41.668999 4840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.3
I20260926 00:59:41.670404 4840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:37913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab
--principal=kudu/127.3.75.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.75.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.75.62:34681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.3
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 00:59:41.670683 4840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 00:59:41.670954 4840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 00:59:41.675750 4846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:41.675762 4845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:41.675805 4848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 00:59:41.676015 4840 server_base.cc:1034] running on GCE node
I20260926 00:59:41.676235 4840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 00:59:41.676482 4840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 00:59:41.677637 4840 hybrid_clock.cc:648] HybridClock initialized: now 1790384381677612 us; error 37 us; skew 500 ppm
Sep 26 00:59:41 dist-test-slave-7v10 krb5kdc[4443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384381, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 00:59:41.680126 4840 init.cc:377] Logged in from keytab as kudu/127.3.75.3@KRBTEST.COM (short username kudu)
I20260926 00:59:41.680608 4840 webserver.cc:460] Webserver started at http://127.3.75.3:45663/ using document root <none> and password file <none>
I20260926 00:59:41.680841 4840 fs_manager.cc:359] Metadata directory not provided
I20260926 00:59:41.680886 4840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 00:59:41.681033 4840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 00:59:41.682102 4840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/ts-2/data/instance:
uuid: "af0947ff9fcc4c6094ab17942c392fff"
format_stamp: "Formatted at 2026-09-26 00:59:41 on dist-test-slave-7v10"
server_key: "d7dfedfb215566a7687af4a5ebe9d4b0"
server_key_iv: "d66781a8908842900d6e64a49b96c784"
server_key_version: "encryptionkey@0"
I20260926 00:59:41.682466 4840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/ts-2/wal/instance:
uuid: "af0947ff9fcc4c6094ab17942c392fff"
format_stamp: "Formatted at 2026-09-26 00:59:41 on dist-test-slave-7v10"
server_key: "d7dfedfb215566a7687af4a5ebe9d4b0"
server_key_iv: "d66781a8908842900d6e64a49b96c784"
server_key_version: "encryptionkey@0"
I20260926 00:59:41.683857 4840 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 00:59:41.684574 4855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:41.684785 4840 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:41.684880 4840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/ts-2/wal
uuid: "af0947ff9fcc4c6094ab17942c392fff"
format_stamp: "Formatted at 2026-09-26 00:59:41 on dist-test-slave-7v10"
server_key: "d7dfedfb215566a7687af4a5ebe9d4b0"
server_key_iv: "d66781a8908842900d6e64a49b96c784"
server_key_version: "encryptionkey@0"
I20260926 00:59:41.684952 4840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 00:59:41.701419 4840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 00:59:41.703347 4840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 00:59:41.703506 4840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 00:59:41.704079 4840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 00:59:41.704119 4840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:41.704141 4840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 00:59:41.704154 4840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 00:59:41 dist-test-slave-7v10 krb5kdc[4443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384381, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.3@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 00:59:41.711510 4840 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.3:43995
I20260926 00:59:41.711598 4970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.3:43995 every 8 connection(s)
I20260926 00:59:41.711902 4840 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
I20260926 00:59:41.714143 4538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:41.704493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.3:55873 (local address 127.3.75.62:34681)
0926 00:59:41.704575 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:41.704577 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:41.704799 (+ 222us) server_negotiation.cc:407] Connection header received
0926 00:59:41.705269 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:41.705270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:41.705284 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:41.705366 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:41.706057 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:41.706635 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:41.707814 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:41.708014 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:41.708098 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:41.711331 (+ 3233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:41.711333 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:41.711333 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 00:59:41.711340 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:41.712837 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:41.713225 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:41.713226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:41.713226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:41.713279 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:41.713529 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:41.713530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:41.713531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:41.713645 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:41.713719 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:41.713974 (+ 255us) server_negotiation.cc:299] Negotiation successful
0926 00:59:41.714058 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 00:59:41.714308 4864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:41.704491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34681 (local address 127.3.75.3:55873)
0926 00:59:41.704662 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:41.704700 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:41.705170 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:41.705463 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:41.705467 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:41.705775 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0926 00:59:41.705959 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:41.705964 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:41.706733 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:41.706735 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:41.707728 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:41.707730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:41.708117 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:41.708118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:41.708228 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:41.708795 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 00:59:41.708806 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 00:59:41.711237 (+ 2431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 00:59:41.712932 (+ 1695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 00:59:41.712938 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 00:59:41.712940 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 00:59:41.713148 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 00:59:41.713350 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 00:59:41.713351 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 00:59:41.713351 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 00:59:41.713457 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 00:59:41.713711 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 00:59:41.713717 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 00:59:41.713851 (+ 134us) client_negotiation.cc:769] Sending connection context
0926 00:59:41.713952 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 00:59:41.714092 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":108,"threads_started":1}
I20260926 00:59:41.714833 4971 heartbeater.cc:347] Connected to a master server at 127.3.75.62:34681
I20260926 00:59:41.714962 4971 heartbeater.cc:464] Registering TS with master...
I20260926 00:59:41.715143 4971 heartbeater.cc:511] Master 127.3.75.62:34681 requested a full tablet report, sending...
I20260926 00:59:41.715490 4487 ts_manager.cc:194] Registered new tserver with Master: af0947ff9fcc4c6094ab17942c392fff (127.3.75.3:43995)
I20260926 00:59:41.715871 4487 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.75.3@KRBTEST.COM'} at 127.3.75.3:55873
I20260926 00:59:41.718248 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 4840
I20260926 00:59:41.718355 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/ts-2/wal/instance
I20260926 00:59:41.718544 3372 external_mini_cluster.cc:1428] Setting key fdf5c7d10b7f4c8d4250de8fc1c3fe9a
I20260926 00:59:41.719563 3372 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 00:59:41.721582 4977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:41.715854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34681 (local address 127.3.75.3:57275)
0926 00:59:41.715989 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:41.716014 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:41.716068 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:41.716231 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:41.716232 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:41.716420 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0926 00:59:41.716479 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:41.716481 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:41.717129 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:41.717129 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:41.717623 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:41.717624 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:41.717941 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:41.717941 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:41.717996 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:41.718420 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 00:59:41.718425 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 00:59:41.718789 (+ 364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 00:59:41.720513 (+ 1724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 00:59:41.720514 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 00:59:41.720514 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 00:59:41.720739 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 00:59:41.720936 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 00:59:41.720937 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 00:59:41.720937 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 00:59:41.721012 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 00:59:41.721233 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 00:59:41.721234 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 00:59:41.721281 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 00:59:41.721358 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 00:59:41.721473 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":73,"threads_started":1}
I20260926 00:59:41.721608 4538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:41.715957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.3:57275 (local address 127.3.75.62:34681)
0926 00:59:41.715997 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:41.715999 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:41.716053 (+ 54us) server_negotiation.cc:407] Connection header received
0926 00:59:41.716123 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:41.716124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:41.716133 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:41.716178 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:41.716567 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:41.717039 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:41.717691 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:41.717865 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:41.717928 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:41.718933 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:41.718934 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:41.718935 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 00:59:41.718942 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:41.720441 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:41.720811 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:41.720811 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:41.720812 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:41.720859 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:41.721072 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:41.721073 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:41.721073 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:41.721176 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:41.721224 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:41.721400 (+ 176us) server_negotiation.cc:299] Negotiation successful
0926 00:59:41.721490 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
Sep 26 00:59:41 dist-test-slave-7v10 krb5kdc[4443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384381, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 00:59:41 dist-test-slave-7v10 krb5kdc[4443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384381, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 00:59:41.743360 4538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:41.736152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47666 (local address 127.3.75.62:34681)
0926 00:59:41.736210 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:41.736211 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:41.736222 (+ 11us) server_negotiation.cc:407] Connection header received
0926 00:59:41.736279 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:41.736280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:41.736288 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:41.736337 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:41.736890 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:41.737368 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:41.738049 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:41.738222 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:41.738270 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:41.740365 (+ 2095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:41.740366 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:41.740366 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 00:59:41.740373 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:41.742229 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:41.742542 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:41.742543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:41.742543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:41.742598 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:41.742760 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:41.742760 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:41.742760 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:41.742864 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:41.743025 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:41.743131 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 00:59:41.743195 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 00:59:41.745182 4487 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47666:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 00:59:41.746745 4487 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 00:59:41.753233 4988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:41.749217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35202 (local address 127.3.75.2:35513)
0926 00:59:41.749389 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:41.749393 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:41.749421 (+ 28us) server_negotiation.cc:407] Connection header received
0926 00:59:41.749459 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:41.749461 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:41.749478 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:41.749910 (+ 432us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 00:59:41.749966 (+ 56us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:41.750679 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:41.751970 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:41.752768 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:41.752843 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:41.752910 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:41.753089 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 00:59:41.753136 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":74,"threads_started":1}
I20260926 00:59:41.753221 4985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:41.749116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.2:35513 (local address 127.0.0.1:35202)
0926 00:59:41.749267 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:41.749309 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:41.749375 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:41.749762 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:41.749765 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:41.749766 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 00:59:41.749832 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:41.749834 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:41.750766 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:41.750767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:41.751895 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:41.751898 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:41.752848 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:41.752849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:41.752944 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:41.752959 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 00:59:41.753069 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 00:59:41.753129 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":2,"thread_start_us":47,"threads_started":1}
I20260926 00:59:41.753512 4986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:41.749119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.3:43995 (local address 127.0.0.1:59224)
0926 00:59:41.749575 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:41.749583 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:41.749635 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:41.749899 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:41.749900 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:41.749902 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 00:59:41.749973 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:41.749976 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:41.751362 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:41.751364 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:41.752278 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:41.752280 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:41.753059 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:41.753060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:41.753124 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:41.753128 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 00:59:41.753174 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 00:59:41.753213 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"mutex_wait_us":23,"thread_start_us":59,"threads_started":1}
I20260926 00:59:41.753615 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:41.749341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59224 (local address 127.3.75.3:43995)
0926 00:59:41.749525 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:41.749530 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:41.749653 (+ 123us) server_negotiation.cc:407] Connection header received
0926 00:59:41.749730 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:41.749732 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:41.749750 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:41.749832 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 00:59:41.750061 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:41.751272 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:41.752363 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:41.752909 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:41.752975 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:41.753025 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:41.753286 (+ 261us) server_negotiation.cc:299] Negotiation successful
0926 00:59:41.753521 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":82,"threads_started":1}
I20260926 00:59:41.754185 4765 tablet_service.cc:1467] Processing CreateTablet for tablet fb57a246128f40489b2c4cd5767a84fc (DEFAULT_TABLE table=test-table [id=bbb9102183f94aa0a4f0c472d4340467]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 00:59:41.754302 4905 tablet_service.cc:1467] Processing CreateTablet for tablet fb57a246128f40489b2c4cd5767a84fc (DEFAULT_TABLE table=test-table [id=bbb9102183f94aa0a4f0c472d4340467]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 00:59:41.754496 4765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb57a246128f40489b2c4cd5767a84fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 00:59:41.754705 4905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb57a246128f40489b2c4cd5767a84fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 00:59:41.757366 4992 tablet_bootstrap.cc:492] T fb57a246128f40489b2c4cd5767a84fc P af0947ff9fcc4c6094ab17942c392fff: Bootstrap starting.
I20260926 00:59:41.757434 4987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:41.749119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.1:37775 (local address 127.0.0.1:40012)
0926 00:59:41.749537 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:41.749548 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:41.749613 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:41.749926 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:41.749927 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:41.749929 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 00:59:41.749964 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:41.749965 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:41.751399 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:41.751400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:41.752668 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:41.752671 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:41.757200 (+ 4529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:41.757201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:41.757267 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:41.757271 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 00:59:41.757319 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 00:59:41.757361 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"spinlock_wait_cycles":768,"thread_start_us":56,"threads_started":1}
I20260926 00:59:41.757457 4991 tablet_bootstrap.cc:492] T fb57a246128f40489b2c4cd5767a84fc P 9066d18b104344189422cfe810192780: Bootstrap starting.
I20260926 00:59:41.757814 4990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:41.749343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40012 (local address 127.3.75.1:37775)
0926 00:59:41.749543 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:41.749547 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:41.749599 (+ 52us) server_negotiation.cc:407] Connection header received
0926 00:59:41.749763 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:41.749766 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:41.749781 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:41.749857 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 00:59:41.750030 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:41.751306 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:41.752763 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:41.757040 (+ 4277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:41.757108 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:41.757164 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:41.757650 (+ 486us) server_negotiation.cc:299] Negotiation successful
0926 00:59:41.757701 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":112,"threads_started":1}
I20260926 00:59:41.758127 4991 tablet_bootstrap.cc:654] T fb57a246128f40489b2c4cd5767a84fc P 9066d18b104344189422cfe810192780: Neither blocks nor log segments found. Creating new log.
I20260926 00:59:41.758472 4991 log.cc:824] T fb57a246128f40489b2c4cd5767a84fc P 9066d18b104344189422cfe810192780: Log is configured to *not* fsync() on all Append() calls
I20260926 00:59:41.758586 4625 tablet_service.cc:1467] Processing CreateTablet for tablet fb57a246128f40489b2c4cd5767a84fc (DEFAULT_TABLE table=test-table [id=bbb9102183f94aa0a4f0c472d4340467]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 00:59:41.758842 4992 tablet_bootstrap.cc:654] T fb57a246128f40489b2c4cd5767a84fc P af0947ff9fcc4c6094ab17942c392fff: Neither blocks nor log segments found. Creating new log.
I20260926 00:59:41.758970 4625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb57a246128f40489b2c4cd5767a84fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 00:59:41.759246 4992 log.cc:824] T fb57a246128f40489b2c4cd5767a84fc P af0947ff9fcc4c6094ab17942c392fff: Log is configured to *not* fsync() on all Append() calls
I20260926 00:59:41.759366 4991 tablet_bootstrap.cc:492] T fb57a246128f40489b2c4cd5767a84fc P 9066d18b104344189422cfe810192780: No bootstrap required, opened a new log
I20260926 00:59:41.759440 4991 ts_tablet_manager.cc:1397] T fb57a246128f40489b2c4cd5767a84fc P 9066d18b104344189422cfe810192780: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 00:59:41.760042 4992 tablet_bootstrap.cc:492] T fb57a246128f40489b2c4cd5767a84fc P af0947ff9fcc4c6094ab17942c392fff: No bootstrap required, opened a new log
I20260926 00:59:41.760114 4992 ts_tablet_manager.cc:1397] T fb57a246128f40489b2c4cd5767a84fc P af0947ff9fcc4c6094ab17942c392fff: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260926 00:59:41.761472 4991 raft_consensus.cc:348] T fb57a246128f40489b2c4cd5767a84fc P 9066d18b104344189422cfe810192780 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507869ed7bf44153a968857556049df0" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37775 } } peers { permanent_uuid: "af0947ff9fcc4c6094ab17942c392fff" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 43995 } } peers { permanent_uuid: "9066d18b104344189422cfe810192780" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 35513 } }
I20260926 00:59:41.761643 4991 raft_consensus.cc:374] T fb57a246128f40489b2c4cd5767a84fc P 9066d18b104344189422cfe810192780 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 00:59:41.761679 4991 raft_consensus.cc:729] T fb57a246128f40489b2c4cd5767a84fc P 9066d18b104344189422cfe810192780 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9066d18b104344189422cfe810192780, State: Initialized, Role: FOLLOWER
I20260926 00:59:41.761822 4991 consensus_queue.cc:260] T fb57a246128f40489b2c4cd5767a84fc P 9066d18b104344189422cfe810192780 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507869ed7bf44153a968857556049df0" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37775 } } peers { permanent_uuid: "af0947ff9fcc4c6094ab17942c392fff" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 43995 } } peers { permanent_uuid: "9066d18b104344189422cfe810192780" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 35513 } }
I20260926 00:59:41.762096 4995 tablet_bootstrap.cc:492] T fb57a246128f40489b2c4cd5767a84fc P 507869ed7bf44153a968857556049df0: Bootstrap starting.
I20260926 00:59:41.762138 4991 ts_tablet_manager.cc:1428] T fb57a246128f40489b2c4cd5767a84fc P 9066d18b104344189422cfe810192780: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 00:59:41.762130 4992 raft_consensus.cc:348] T fb57a246128f40489b2c4cd5767a84fc P af0947ff9fcc4c6094ab17942c392fff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507869ed7bf44153a968857556049df0" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37775 } } peers { permanent_uuid: "af0947ff9fcc4c6094ab17942c392fff" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 43995 } } peers { permanent_uuid: "9066d18b104344189422cfe810192780" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 35513 } }
I20260926 00:59:41.762272 4831 heartbeater.cc:503] Master 127.3.75.62:34681 was elected leader, sending a full tablet report...
I20260926 00:59:41.762279 4992 raft_consensus.cc:374] T fb57a246128f40489b2c4cd5767a84fc P af0947ff9fcc4c6094ab17942c392fff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 00:59:41.762331 4992 raft_consensus.cc:729] T fb57a246128f40489b2c4cd5767a84fc P af0947ff9fcc4c6094ab17942c392fff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af0947ff9fcc4c6094ab17942c392fff, State: Initialized, Role: FOLLOWER
I20260926 00:59:41.762452 4992 consensus_queue.cc:260] T fb57a246128f40489b2c4cd5767a84fc P af0947ff9fcc4c6094ab17942c392fff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507869ed7bf44153a968857556049df0" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37775 } } peers { permanent_uuid: "af0947ff9fcc4c6094ab17942c392fff" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 43995 } } peers { permanent_uuid: "9066d18b104344189422cfe810192780" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 35513 } }
I20260926 00:59:41.762740 4971 heartbeater.cc:503] Master 127.3.75.62:34681 was elected leader, sending a full tablet report...
I20260926 00:59:41.762771 4995 tablet_bootstrap.cc:654] T fb57a246128f40489b2c4cd5767a84fc P 507869ed7bf44153a968857556049df0: Neither blocks nor log segments found. Creating new log.
I20260926 00:59:41.762724 4992 ts_tablet_manager.cc:1428] T fb57a246128f40489b2c4cd5767a84fc P af0947ff9fcc4c6094ab17942c392fff: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260926 00:59:41.763089 4995 log.cc:824] T fb57a246128f40489b2c4cd5767a84fc P 507869ed7bf44153a968857556049df0: Log is configured to *not* fsync() on all Append() calls
I20260926 00:59:41.763981 4995 tablet_bootstrap.cc:492] T fb57a246128f40489b2c4cd5767a84fc P 507869ed7bf44153a968857556049df0: No bootstrap required, opened a new log
I20260926 00:59:41.764070 4995 ts_tablet_manager.cc:1397] T fb57a246128f40489b2c4cd5767a84fc P 507869ed7bf44153a968857556049df0: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 00:59:41.765540 4995 raft_consensus.cc:348] T fb57a246128f40489b2c4cd5767a84fc P 507869ed7bf44153a968857556049df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507869ed7bf44153a968857556049df0" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37775 } } peers { permanent_uuid: "af0947ff9fcc4c6094ab17942c392fff" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 43995 } } peers { permanent_uuid: "9066d18b104344189422cfe810192780" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 35513 } }
I20260926 00:59:41.765651 4995 raft_consensus.cc:374] T fb57a246128f40489b2c4cd5767a84fc P 507869ed7bf44153a968857556049df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 00:59:41.765671 4995 raft_consensus.cc:729] T fb57a246128f40489b2c4cd5767a84fc P 507869ed7bf44153a968857556049df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 507869ed7bf44153a968857556049df0, State: Initialized, Role: FOLLOWER
I20260926 00:59:41.765738 4995 consensus_queue.cc:260] T fb57a246128f40489b2c4cd5767a84fc P 507869ed7bf44153a968857556049df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507869ed7bf44153a968857556049df0" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37775 } } peers { permanent_uuid: "af0947ff9fcc4c6094ab17942c392fff" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 43995 } } peers { permanent_uuid: "9066d18b104344189422cfe810192780" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 35513 } }
I20260926 00:59:41.765954 4995 ts_tablet_manager.cc:1428] T fb57a246128f40489b2c4cd5767a84fc P 507869ed7bf44153a968857556049df0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 00:59:41.766014 4691 heartbeater.cc:503] Master 127.3.75.62:34681 was elected leader, sending a full tablet report...
W20260926 00:59:41.805764 4832 tablet.cc:2369] T fb57a246128f40489b2c4cd5767a84fc P 9066d18b104344189422cfe810192780: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 00:59:41.897255 4692 tablet.cc:2369] T fb57a246128f40489b2c4cd5767a84fc P 507869ed7bf44153a968857556049df0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 00:59:41.905180 4997 raft_consensus.cc:482] T fb57a246128f40489b2c4cd5767a84fc P af0947ff9fcc4c6094ab17942c392fff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 00:59:41.905290 4997 raft_consensus.cc:504] T fb57a246128f40489b2c4cd5767a84fc P af0947ff9fcc4c6094ab17942c392fff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507869ed7bf44153a968857556049df0" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37775 } } peers { permanent_uuid: "af0947ff9fcc4c6094ab17942c392fff" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 43995 } } peers { permanent_uuid: "9066d18b104344189422cfe810192780" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 35513 } }
I20260926 00:59:41.905610 4997 leader_election.cc:288] T fb57a246128f40489b2c4cd5767a84fc P af0947ff9fcc4c6094ab17942c392fff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 507869ed7bf44153a968857556049df0 (127.3.75.1:37775), 9066d18b104344189422cfe810192780 (127.3.75.2:35513)
I20260926 00:59:41.907439 4996 raft_consensus.cc:482] T fb57a246128f40489b2c4cd5767a84fc P 9066d18b104344189422cfe810192780 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 00:59:41.907578 4996 raft_consensus.cc:504] T fb57a246128f40489b2c4cd5767a84fc P 9066d18b104344189422cfe810192780 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507869ed7bf44153a968857556049df0" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37775 } } peers { permanent_uuid: "af0947ff9fcc4c6094ab17942c392fff" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 43995 } } peers { permanent_uuid: "9066d18b104344189422cfe810192780" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 35513 } }
I20260926 00:59:41.907931 4996 leader_election.cc:288] T fb57a246128f40489b2c4cd5767a84fc P 9066d18b104344189422cfe810192780 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 507869ed7bf44153a968857556049df0 (127.3.75.1:37775), af0947ff9fcc4c6094ab17942c392fff (127.3.75.3:43995)
I20260926 00:59:41.909672 4864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:41.905836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.2:35513 (local address 127.3.75.3:58223)
0926 00:59:41.905887 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:41.905901 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:41.905947 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:41.906142 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:41.906144 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:41.906146 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 00:59:41.906224 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:41.906226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:41.907215 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:41.907216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:41.908524 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:41.908527 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:41.909405 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:41.909406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:41.909487 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:41.909492 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 00:59:41.909551 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 00:59:41.909596 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":28}
I20260926 00:59:41.909858 4988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:41.905912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.3:58223 (local address 127.3.75.2:35513)
0926 00:59:41.905969 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:41.905972 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:41.905984 (+ 12us) server_negotiation.cc:407] Connection header received
0926 00:59:41.906016 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:41.906017 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:41.906044 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:41.906104 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 00:59:41.906310 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:41.907132 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:41.908625 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:41.909205 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:41.909268 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:41.909279 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:41.909768 (+ 489us) server_negotiation.cc:299] Negotiation successful
0926 00:59:41.909802 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 00:59:41.909869 4990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:41.905947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.3:37235 (local address 127.3.75.1:37775)
0926 00:59:41.905991 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:41.905993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:41.906289 (+ 296us) server_negotiation.cc:407] Connection header received
0926 00:59:41.906324 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:41.906325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:41.906335 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:41.906389 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 00:59:41.906523 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:41.907238 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:41.908828 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:41.909379 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:41.909426 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:41.909437 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:41.909655 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 00:59:41.909703 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 00:59:41.910107 5000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:41.905836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.1:37775 (local address 127.3.75.3:37235)
0926 00:59:41.906171 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:41.906182 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:41.906277 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:41.906398 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:41.906399 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:41.906400 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 00:59:41.906452 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:41.906453 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:41.907321 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:41.907322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:41.908745 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:41.908746 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:41.909495 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:41.909496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:41.909574 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:41.909577 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 00:59:41.909974 (+ 397us) client_negotiation.cc:240] Negotiation successful
0926 00:59:41.910014 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":306,"thread_start_us":104,"threads_started":1}
I20260926 00:59:41.910398 4785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb57a246128f40489b2c4cd5767a84fc" candidate_uuid: "af0947ff9fcc4c6094ab17942c392fff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9066d18b104344189422cfe810192780" is_pre_election: true
I20260926 00:59:41.910513 4785 raft_consensus.cc:2454] T fb57a246128f40489b2c4cd5767a84fc P 9066d18b104344189422cfe810192780 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af0947ff9fcc4c6094ab17942c392fff in term 0.
I20260926 00:59:41.910480 4645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb57a246128f40489b2c4cd5767a84fc" candidate_uuid: "af0947ff9fcc4c6094ab17942c392fff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "507869ed7bf44153a968857556049df0" is_pre_election: true
I20260926 00:59:41.910604 4645 raft_consensus.cc:2454] T fb57a246128f40489b2c4cd5767a84fc P 507869ed7bf44153a968857556049df0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af0947ff9fcc4c6094ab17942c392fff in term 0.
I20260926 00:59:41.910820 4859 leader_election.cc:302] T fb57a246128f40489b2c4cd5767a84fc P af0947ff9fcc4c6094ab17942c392fff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 507869ed7bf44153a968857556049df0, af0947ff9fcc4c6094ab17942c392fff; no voters:
I20260926 00:59:41.911048 4997 raft_consensus.cc:2781] T fb57a246128f40489b2c4cd5767a84fc P af0947ff9fcc4c6094ab17942c392fff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 00:59:41.911140 4997 raft_consensus.cc:482] T fb57a246128f40489b2c4cd5767a84fc P af0947ff9fcc4c6094ab17942c392fff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 00:59:41.911176 4997 raft_consensus.cc:3037] T fb57a246128f40489b2c4cd5767a84fc P af0947ff9fcc4c6094ab17942c392fff [term 0 FOLLOWER]: Advancing to term 1
I20260926 00:59:41.911794 5002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:41.908246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:53501 (local address 127.3.75.1:37775)
0926 00:59:41.908508 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:41.908510 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:41.908518 (+ 8us) server_negotiation.cc:407] Connection header received
0926 00:59:41.908541 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:41.908541 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:41.908552 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:41.908760 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 00:59:41.908804 (+ 44us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:41.909481 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:41.910899 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:41.911403 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:41.911499 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:41.911509 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:41.911673 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 00:59:41.911734 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":60,"threads_started":1}
I20260926 00:59:41.911785 4724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:41.908038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.1:37775 (local address 127.3.75.2:53501)
0926 00:59:41.908122 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:41.908133 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:41.908172 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:41.908643 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:41.908644 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:41.908646 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 00:59:41.908699 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:41.908700 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:41.909578 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:41.909579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:41.910745 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:41.910747 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:41.911507 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:41.911508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:41.911594 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:41.911598 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 00:59:41.911649 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 00:59:41.911697 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68,"mutex_wait_us":1}
I20260926 00:59:41.911947 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:41.908288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:59551 (local address 127.3.75.3:43995)
0926 00:59:41.908325 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:41.908327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:41.908394 (+ 67us) server_negotiation.cc:407] Connection header received
0926 00:59:41.908453 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:41.908453 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:41.908461 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:41.908509 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 00:59:41.908701 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:41.909385 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:41.910906 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:41.911468 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:41.911552 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:41.911561 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:41.911830 (+ 269us) server_negotiation.cc:299] Negotiation successful
0926 00:59:41.911878 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 00:59:41.912068 4645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb57a246128f40489b2c4cd5767a84fc" candidate_uuid: "9066d18b104344189422cfe810192780" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "507869ed7bf44153a968857556049df0" is_pre_election: true
I20260926 00:59:41.911803 5001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:41.908100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.3:43995 (local address 127.3.75.2:59551)
0926 00:59:41.908331 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:41.908342 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:41.908406 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:41.908557 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:41.908558 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:41.908560 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 00:59:41.908625 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:41.908626 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:41.909471 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:41.909472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:41.910765 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:41.910768 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:41.911556 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:41.911557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:41.911629 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:41.911633 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 00:59:41.911681 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 00:59:41.911707 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":120,"thread_start_us":105,"threads_started":1}
I20260926 00:59:41.912142 4645 raft_consensus.cc:2454] T fb57a246128f40489b2c4cd5767a84fc P 507869ed7bf44153a968857556049df0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9066d18b104344189422cfe810192780 in term 0.
I20260926 00:59:41.912325 4719 leader_election.cc:302] T fb57a246128f40489b2c4cd5767a84fc P 9066d18b104344189422cfe810192780 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 507869ed7bf44153a968857556049df0, 9066d18b104344189422cfe810192780; no voters:
I20260926 00:59:41.912333 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb57a246128f40489b2c4cd5767a84fc" candidate_uuid: "9066d18b104344189422cfe810192780" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af0947ff9fcc4c6094ab17942c392fff" is_pre_election: true
I20260926 00:59:41.912456 4996 raft_consensus.cc:2781] T fb57a246128f40489b2c4cd5767a84fc P 9066d18b104344189422cfe810192780 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 00:59:41.912509 4996 raft_consensus.cc:482] T fb57a246128f40489b2c4cd5767a84fc P 9066d18b104344189422cfe810192780 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 00:59:41.912554 4996 raft_consensus.cc:3037] T fb57a246128f40489b2c4cd5767a84fc P 9066d18b104344189422cfe810192780 [term 0 FOLLOWER]: Advancing to term 1
I20260926 00:59:41.912638 4997 raft_consensus.cc:504] T fb57a246128f40489b2c4cd5767a84fc P af0947ff9fcc4c6094ab17942c392fff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507869ed7bf44153a968857556049df0" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37775 } } peers { permanent_uuid: "af0947ff9fcc4c6094ab17942c392fff" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 43995 } } peers { permanent_uuid: "9066d18b104344189422cfe810192780" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 35513 } }
I20260926 00:59:41.912791 4925 raft_consensus.cc:2379] T fb57a246128f40489b2c4cd5767a84fc P af0947ff9fcc4c6094ab17942c392fff [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9066d18b104344189422cfe810192780 in current term 1: Already voted for candidate af0947ff9fcc4c6094ab17942c392fff in this term.
I20260926 00:59:41.912923 4645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb57a246128f40489b2c4cd5767a84fc" candidate_uuid: "af0947ff9fcc4c6094ab17942c392fff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "507869ed7bf44153a968857556049df0"
I20260926 00:59:41.912961 4785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb57a246128f40489b2c4cd5767a84fc" candidate_uuid: "af0947ff9fcc4c6094ab17942c392fff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9066d18b104344189422cfe810192780"
I20260926 00:59:41.913010 4645 raft_consensus.cc:3037] T fb57a246128f40489b2c4cd5767a84fc P 507869ed7bf44153a968857556049df0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 00:59:41.913130 4997 leader_election.cc:288] T fb57a246128f40489b2c4cd5767a84fc P af0947ff9fcc4c6094ab17942c392fff [CANDIDATE]: Term 1 election: Requested vote from peers 507869ed7bf44153a968857556049df0 (127.3.75.1:37775), 9066d18b104344189422cfe810192780 (127.3.75.2:35513)
I20260926 00:59:41.913385 4996 raft_consensus.cc:504] T fb57a246128f40489b2c4cd5767a84fc P 9066d18b104344189422cfe810192780 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507869ed7bf44153a968857556049df0" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37775 } } peers { permanent_uuid: "af0947ff9fcc4c6094ab17942c392fff" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 43995 } } peers { permanent_uuid: "9066d18b104344189422cfe810192780" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 35513 } }
I20260926 00:59:41.913488 4785 raft_consensus.cc:2379] T fb57a246128f40489b2c4cd5767a84fc P 9066d18b104344189422cfe810192780 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate af0947ff9fcc4c6094ab17942c392fff in current term 1: Already voted for candidate 9066d18b104344189422cfe810192780 in this term.
I20260926 00:59:41.913537 4996 leader_election.cc:288] T fb57a246128f40489b2c4cd5767a84fc P 9066d18b104344189422cfe810192780 [CANDIDATE]: Term 1 election: Requested vote from peers 507869ed7bf44153a968857556049df0 (127.3.75.1:37775), af0947ff9fcc4c6094ab17942c392fff (127.3.75.3:43995)
I20260926 00:59:41.913674 4644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb57a246128f40489b2c4cd5767a84fc" candidate_uuid: "9066d18b104344189422cfe810192780" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "507869ed7bf44153a968857556049df0"
I20260926 00:59:41.913793 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb57a246128f40489b2c4cd5767a84fc" candidate_uuid: "9066d18b104344189422cfe810192780" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af0947ff9fcc4c6094ab17942c392fff"
I20260926 00:59:41.913854 4925 raft_consensus.cc:2379] T fb57a246128f40489b2c4cd5767a84fc P af0947ff9fcc4c6094ab17942c392fff [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9066d18b104344189422cfe810192780 in current term 1: Already voted for candidate af0947ff9fcc4c6094ab17942c392fff in this term.
I20260926 00:59:41.913856 4645 raft_consensus.cc:2454] T fb57a246128f40489b2c4cd5767a84fc P 507869ed7bf44153a968857556049df0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af0947ff9fcc4c6094ab17942c392fff in term 1.
I20260926 00:59:41.914018 4716 leader_election.cc:302] T fb57a246128f40489b2c4cd5767a84fc P 9066d18b104344189422cfe810192780 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9066d18b104344189422cfe810192780; no voters: 507869ed7bf44153a968857556049df0, af0947ff9fcc4c6094ab17942c392fff
I20260926 00:59:41.914032 4859 leader_election.cc:302] T fb57a246128f40489b2c4cd5767a84fc P af0947ff9fcc4c6094ab17942c392fff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 507869ed7bf44153a968857556049df0, af0947ff9fcc4c6094ab17942c392fff; no voters: 9066d18b104344189422cfe810192780
I20260926 00:59:41.914112 4996 raft_consensus.cc:2726] T fb57a246128f40489b2c4cd5767a84fc P 9066d18b104344189422cfe810192780 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260926 00:59:41.914124 4997 raft_consensus.cc:2781] T fb57a246128f40489b2c4cd5767a84fc P af0947ff9fcc4c6094ab17942c392fff [term 1 FOLLOWER]: Leader election won for term 1
I20260926 00:59:41.914249 4997 raft_consensus.cc:686] T fb57a246128f40489b2c4cd5767a84fc P af0947ff9fcc4c6094ab17942c392fff [term 1 LEADER]: Becoming Leader. State: Replica: af0947ff9fcc4c6094ab17942c392fff, State: Running, Role: LEADER
I20260926 00:59:41.914351 4997 consensus_queue.cc:237] T fb57a246128f40489b2c4cd5767a84fc P af0947ff9fcc4c6094ab17942c392fff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507869ed7bf44153a968857556049df0" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37775 } } peers { permanent_uuid: "af0947ff9fcc4c6094ab17942c392fff" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 43995 } } peers { permanent_uuid: "9066d18b104344189422cfe810192780" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 35513 } }
I20260926 00:59:41.915112 4487 catalog_manager.cc:5495] T fb57a246128f40489b2c4cd5767a84fc P af0947ff9fcc4c6094ab17942c392fff reported cstate change: term changed from 0 to 1, leader changed from <none> to af0947ff9fcc4c6094ab17942c392fff (127.3.75.3). New cstate: current_term: 1 leader_uuid: "af0947ff9fcc4c6094ab17942c392fff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507869ed7bf44153a968857556049df0" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af0947ff9fcc4c6094ab17942c392fff" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 43995 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9066d18b104344189422cfe810192780" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 35513 } health_report { overall_health: UNKNOWN } } }
I20260926 00:59:41.934773 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:41.932100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59232 (local address 127.3.75.3:43995)
0926 00:59:41.932158 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:41.932160 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:41.932168 (+ 8us) server_negotiation.cc:407] Connection header received
0926 00:59:41.932274 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:41.932275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:41.932292 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:41.932369 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 00:59:41.932524 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:41.932971 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:41.933858 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:41.934050 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:41.934131 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:41.934396 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 00:59:41.934494 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 00:59:41.934587 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:41.934642 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 00:59:41.934662 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 00:59:41.936609 4785 raft_consensus.cc:1261] T fb57a246128f40489b2c4cd5767a84fc P 9066d18b104344189422cfe810192780 [term 1 FOLLOWER]: Refusing update from remote peer af0947ff9fcc4c6094ab17942c392fff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 00:59:41.936631 4645 raft_consensus.cc:1261] T fb57a246128f40489b2c4cd5767a84fc P 507869ed7bf44153a968857556049df0 [term 1 FOLLOWER]: Refusing update from remote peer af0947ff9fcc4c6094ab17942c392fff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 00:59:41.936852 4997 consensus_queue.cc:1035] T fb57a246128f40489b2c4cd5767a84fc P af0947ff9fcc4c6094ab17942c392fff [LEADER]: Connected to new peer: Peer: permanent_uuid: "507869ed7bf44153a968857556049df0" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 00:59:41.937009 5003 consensus_queue.cc:1035] T fb57a246128f40489b2c4cd5767a84fc P af0947ff9fcc4c6094ab17942c392fff [LEADER]: Connected to new peer: Peer: permanent_uuid: "9066d18b104344189422cfe810192780" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 35513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 00:59:41.939383 5008 mvcc.cc:205] Tried to move back new op lower bound from 7333414428409745408 to 7333414428321751040. Current Snapshot: MvccSnapshot[applied={T|T < 7333414428409745408}]
I20260926 00:59:41.939445 5007 mvcc.cc:205] Tried to move back new op lower bound from 7333414428409745408 to 7333414428321751040. Current Snapshot: MvccSnapshot[applied={T|T < 7333414428409745408}]
I20260926 00:59:41.939746 5006 mvcc.cc:205] Tried to move back new op lower bound from 7333414428409745408 to 7333414428321751040. Current Snapshot: MvccSnapshot[applied={T|T < 7333414428409745408 or (T in {7333414428409745408})}]
I20260926 00:59:41.942631 4488 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 00:59:41.942726 4488 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 00:59:41.943521 4488 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 03a7c52ba87c4b0a875dec905ad499a1: Sending DeleteTablet for 3 replicas of tablet fb57a246128f40489b2c4cd5767a84fc
I20260926 00:59:41.943859 4625 tablet_service.cc:1514] Processing DeleteTablet for tablet fb57a246128f40489b2c4cd5767a84fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 00:59:41 UTC) from {username='kudu', principal='kudu/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:40012
I20260926 00:59:41.943861 4905 tablet_service.cc:1514] Processing DeleteTablet for tablet fb57a246128f40489b2c4cd5767a84fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 00:59:41 UTC) from {username='kudu', principal='kudu/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:59224
I20260926 00:59:41.943861 4765 tablet_service.cc:1514] Processing DeleteTablet for tablet fb57a246128f40489b2c4cd5767a84fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 00:59:41 UTC) from {username='kudu', principal='kudu/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:35202
I20260926 00:59:41.944103 5018 tablet_replica.cc:330] T fb57a246128f40489b2c4cd5767a84fc P 507869ed7bf44153a968857556049df0: stopping tablet replica
I20260926 00:59:41.944211 5018 raft_consensus.cc:2229] T fb57a246128f40489b2c4cd5767a84fc P 507869ed7bf44153a968857556049df0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 00:59:41.944304 5018 raft_consensus.cc:2258] T fb57a246128f40489b2c4cd5767a84fc P 507869ed7bf44153a968857556049df0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 00:59:41.944420 4488 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47666:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 00:59:41.944566 4488 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 00:59:41.944718 5017 tablet_replica.cc:330] T fb57a246128f40489b2c4cd5767a84fc P af0947ff9fcc4c6094ab17942c392fff: stopping tablet replica
I20260926 00:59:41.944728 5018 ts_tablet_manager.cc:1905] T fb57a246128f40489b2c4cd5767a84fc P 507869ed7bf44153a968857556049df0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 00:59:41.944872 5017 raft_consensus.cc:2229] T fb57a246128f40489b2c4cd5767a84fc P af0947ff9fcc4c6094ab17942c392fff [term 1 LEADER]: Raft consensus shutting down.
I20260926 00:59:41.945020 5017 raft_consensus.cc:2258] T fb57a246128f40489b2c4cd5767a84fc P af0947ff9fcc4c6094ab17942c392fff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 00:59:41.945495 5017 ts_tablet_manager.cc:1905] T fb57a246128f40489b2c4cd5767a84fc P af0947ff9fcc4c6094ab17942c392fff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 00:59:41.945734 4905 tablet_service.cc:1467] Processing CreateTablet for tablet fb81ade8414540ebb01fda61884d1b3c (DEFAULT_TABLE table=test-table [id=21f5e2e32b204f0bbf74486af94df07f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 00:59:41.945874 4905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb81ade8414540ebb01fda61884d1b3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 00:59:41.946463 5016 tablet_replica.cc:330] T fb57a246128f40489b2c4cd5767a84fc P 9066d18b104344189422cfe810192780: stopping tablet replica
I20260926 00:59:41.946566 5016 raft_consensus.cc:2229] T fb57a246128f40489b2c4cd5767a84fc P 9066d18b104344189422cfe810192780 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 00:59:41.946681 5016 raft_consensus.cc:2258] T fb57a246128f40489b2c4cd5767a84fc P 9066d18b104344189422cfe810192780 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 00:59:41.946691 5018 ts_tablet_manager.cc:1918] T fb57a246128f40489b2c4cd5767a84fc P 507869ed7bf44153a968857556049df0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 00:59:41.946753 5018 log.cc:1196] T fb57a246128f40489b2c4cd5767a84fc P 507869ed7bf44153a968857556049df0: Deleting WAL directory at /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/ts-0/wal/wals/fb57a246128f40489b2c4cd5767a84fc
I20260926 00:59:41.947052 5018 ts_tablet_manager.cc:1939] T fb57a246128f40489b2c4cd5767a84fc P 507869ed7bf44153a968857556049df0: Deleting consensus metadata
I20260926 00:59:41.947172 5016 ts_tablet_manager.cc:1905] T fb57a246128f40489b2c4cd5767a84fc P 9066d18b104344189422cfe810192780: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 00:59:41.947299 4992 tablet_bootstrap.cc:492] T fb81ade8414540ebb01fda61884d1b3c P af0947ff9fcc4c6094ab17942c392fff: Bootstrap starting.
I20260926 00:59:41.947690 4992 tablet_bootstrap.cc:654] T fb81ade8414540ebb01fda61884d1b3c P af0947ff9fcc4c6094ab17942c392fff: Neither blocks nor log segments found. Creating new log.
I20260926 00:59:41.947808 5017 ts_tablet_manager.cc:1918] T fb57a246128f40489b2c4cd5767a84fc P af0947ff9fcc4c6094ab17942c392fff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 00:59:41.947919 5017 log.cc:1196] T fb57a246128f40489b2c4cd5767a84fc P af0947ff9fcc4c6094ab17942c392fff: Deleting WAL directory at /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/ts-2/wal/wals/fb57a246128f40489b2c4cd5767a84fc
I20260926 00:59:41.948199 4765 tablet_service.cc:1467] Processing CreateTablet for tablet fb81ade8414540ebb01fda61884d1b3c (DEFAULT_TABLE table=test-table [id=21f5e2e32b204f0bbf74486af94df07f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 00:59:41.948231 5017 ts_tablet_manager.cc:1939] T fb57a246128f40489b2c4cd5767a84fc P af0947ff9fcc4c6094ab17942c392fff: Deleting consensus metadata
I20260926 00:59:41.948312 4765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb81ade8414540ebb01fda61884d1b3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 00:59:41.948701 4472 catalog_manager.cc:4841] TS af0947ff9fcc4c6094ab17942c392fff (127.3.75.3:43995): tablet fb57a246128f40489b2c4cd5767a84fc (table test-table [id=bbb9102183f94aa0a4f0c472d4340467]) successfully deleted
I20260926 00:59:41.948779 5016 ts_tablet_manager.cc:1918] T fb57a246128f40489b2c4cd5767a84fc P 9066d18b104344189422cfe810192780: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 00:59:41.948843 5016 log.cc:1196] T fb57a246128f40489b2c4cd5767a84fc P 9066d18b104344189422cfe810192780: Deleting WAL directory at /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/ts-1/wal/wals/fb57a246128f40489b2c4cd5767a84fc
I20260926 00:59:41.949100 5016 ts_tablet_manager.cc:1939] T fb57a246128f40489b2c4cd5767a84fc P 9066d18b104344189422cfe810192780: Deleting consensus metadata
I20260926 00:59:41.949457 4474 catalog_manager.cc:4841] TS 9066d18b104344189422cfe810192780 (127.3.75.2:35513): tablet fb57a246128f40489b2c4cd5767a84fc (table test-table [id=bbb9102183f94aa0a4f0c472d4340467]) successfully deleted
I20260926 00:59:41.949502 4991 tablet_bootstrap.cc:492] T fb81ade8414540ebb01fda61884d1b3c P 9066d18b104344189422cfe810192780: Bootstrap starting.
I20260926 00:59:41.949599 4475 catalog_manager.cc:4841] TS 507869ed7bf44153a968857556049df0 (127.3.75.1:37775): tablet fb57a246128f40489b2c4cd5767a84fc (table test-table [id=bbb9102183f94aa0a4f0c472d4340467]) successfully deleted
I20260926 00:59:41.949663 4625 tablet_service.cc:1467] Processing CreateTablet for tablet fb81ade8414540ebb01fda61884d1b3c (DEFAULT_TABLE table=test-table [id=21f5e2e32b204f0bbf74486af94df07f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 00:59:41.949772 4625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb81ade8414540ebb01fda61884d1b3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 00:59:41.949895 4991 tablet_bootstrap.cc:654] T fb81ade8414540ebb01fda61884d1b3c P 9066d18b104344189422cfe810192780: Neither blocks nor log segments found. Creating new log.
I20260926 00:59:41.950337 4992 tablet_bootstrap.cc:492] T fb81ade8414540ebb01fda61884d1b3c P af0947ff9fcc4c6094ab17942c392fff: No bootstrap required, opened a new log
I20260926 00:59:41.950388 4992 ts_tablet_manager.cc:1397] T fb81ade8414540ebb01fda61884d1b3c P af0947ff9fcc4c6094ab17942c392fff: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260926 00:59:41.950520 4992 raft_consensus.cc:348] T fb81ade8414540ebb01fda61884d1b3c P af0947ff9fcc4c6094ab17942c392fff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507869ed7bf44153a968857556049df0" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37775 } } peers { permanent_uuid: "9066d18b104344189422cfe810192780" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 35513 } } peers { permanent_uuid: "af0947ff9fcc4c6094ab17942c392fff" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 43995 } }
I20260926 00:59:41.950587 4992 raft_consensus.cc:374] T fb81ade8414540ebb01fda61884d1b3c P af0947ff9fcc4c6094ab17942c392fff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 00:59:41.950610 4992 raft_consensus.cc:729] T fb81ade8414540ebb01fda61884d1b3c P af0947ff9fcc4c6094ab17942c392fff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af0947ff9fcc4c6094ab17942c392fff, State: Initialized, Role: FOLLOWER
I20260926 00:59:41.950649 4992 consensus_queue.cc:260] T fb81ade8414540ebb01fda61884d1b3c P af0947ff9fcc4c6094ab17942c392fff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507869ed7bf44153a968857556049df0" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37775 } } peers { permanent_uuid: "9066d18b104344189422cfe810192780" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 35513 } } peers { permanent_uuid: "af0947ff9fcc4c6094ab17942c392fff" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 43995 } }
I20260926 00:59:41.950769 4992 ts_tablet_manager.cc:1428] T fb81ade8414540ebb01fda61884d1b3c P af0947ff9fcc4c6094ab17942c392fff: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 00:59:41.950979 4995 tablet_bootstrap.cc:492] T fb81ade8414540ebb01fda61884d1b3c P 507869ed7bf44153a968857556049df0: Bootstrap starting.
I20260926 00:59:41.951403 4991 tablet_bootstrap.cc:492] T fb81ade8414540ebb01fda61884d1b3c P 9066d18b104344189422cfe810192780: No bootstrap required, opened a new log
I20260926 00:59:41.951443 4991 ts_tablet_manager.cc:1397] T fb81ade8414540ebb01fda61884d1b3c P 9066d18b104344189422cfe810192780: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 00:59:41.951578 4991 raft_consensus.cc:348] T fb81ade8414540ebb01fda61884d1b3c P 9066d18b104344189422cfe810192780 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507869ed7bf44153a968857556049df0" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37775 } } peers { permanent_uuid: "9066d18b104344189422cfe810192780" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 35513 } } peers { permanent_uuid: "af0947ff9fcc4c6094ab17942c392fff" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 43995 } }
I20260926 00:59:41.951658 4995 tablet_bootstrap.cc:654] T fb81ade8414540ebb01fda61884d1b3c P 507869ed7bf44153a968857556049df0: Neither blocks nor log segments found. Creating new log.
I20260926 00:59:41.951665 4991 raft_consensus.cc:374] T fb81ade8414540ebb01fda61884d1b3c P 9066d18b104344189422cfe810192780 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 00:59:41.951689 4991 raft_consensus.cc:729] T fb81ade8414540ebb01fda61884d1b3c P 9066d18b104344189422cfe810192780 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9066d18b104344189422cfe810192780, State: Initialized, Role: FOLLOWER
I20260926 00:59:41.951745 4991 consensus_queue.cc:260] T fb81ade8414540ebb01fda61884d1b3c P 9066d18b104344189422cfe810192780 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507869ed7bf44153a968857556049df0" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37775 } } peers { permanent_uuid: "9066d18b104344189422cfe810192780" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 35513 } } peers { permanent_uuid: "af0947ff9fcc4c6094ab17942c392fff" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 43995 } }
I20260926 00:59:41.951843 4991 ts_tablet_manager.cc:1428] T fb81ade8414540ebb01fda61884d1b3c P 9066d18b104344189422cfe810192780: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:41.952229 4995 tablet_bootstrap.cc:492] T fb81ade8414540ebb01fda61884d1b3c P 507869ed7bf44153a968857556049df0: No bootstrap required, opened a new log
I20260926 00:59:41.952276 4995 ts_tablet_manager.cc:1397] T fb81ade8414540ebb01fda61884d1b3c P 507869ed7bf44153a968857556049df0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 00:59:41.952410 4995 raft_consensus.cc:348] T fb81ade8414540ebb01fda61884d1b3c P 507869ed7bf44153a968857556049df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507869ed7bf44153a968857556049df0" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37775 } } peers { permanent_uuid: "9066d18b104344189422cfe810192780" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 35513 } } peers { permanent_uuid: "af0947ff9fcc4c6094ab17942c392fff" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 43995 } }
I20260926 00:59:41.952497 4995 raft_consensus.cc:374] T fb81ade8414540ebb01fda61884d1b3c P 507869ed7bf44153a968857556049df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 00:59:41.952530 4995 raft_consensus.cc:729] T fb81ade8414540ebb01fda61884d1b3c P 507869ed7bf44153a968857556049df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 507869ed7bf44153a968857556049df0, State: Initialized, Role: FOLLOWER
I20260926 00:59:41.952579 4995 consensus_queue.cc:260] T fb81ade8414540ebb01fda61884d1b3c P 507869ed7bf44153a968857556049df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507869ed7bf44153a968857556049df0" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37775 } } peers { permanent_uuid: "9066d18b104344189422cfe810192780" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 35513 } } peers { permanent_uuid: "af0947ff9fcc4c6094ab17942c392fff" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 43995 } }
I20260926 00:59:41.952661 4995 ts_tablet_manager.cc:1428] T fb81ade8414540ebb01fda61884d1b3c P 507869ed7bf44153a968857556049df0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260926 00:59:41.962626 4972 tablet.cc:2369] T fb81ade8414540ebb01fda61884d1b3c P af0947ff9fcc4c6094ab17942c392fff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 00:59:42.015264 5011 raft_consensus.cc:482] T fb81ade8414540ebb01fda61884d1b3c P af0947ff9fcc4c6094ab17942c392fff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 00:59:42.015383 5011 raft_consensus.cc:504] T fb81ade8414540ebb01fda61884d1b3c P af0947ff9fcc4c6094ab17942c392fff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507869ed7bf44153a968857556049df0" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37775 } } peers { permanent_uuid: "9066d18b104344189422cfe810192780" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 35513 } } peers { permanent_uuid: "af0947ff9fcc4c6094ab17942c392fff" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 43995 } }
I20260926 00:59:42.015592 5011 leader_election.cc:288] T fb81ade8414540ebb01fda61884d1b3c P af0947ff9fcc4c6094ab17942c392fff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 507869ed7bf44153a968857556049df0 (127.3.75.1:37775), 9066d18b104344189422cfe810192780 (127.3.75.2:35513)
I20260926 00:59:42.015758 4645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb81ade8414540ebb01fda61884d1b3c" candidate_uuid: "af0947ff9fcc4c6094ab17942c392fff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "507869ed7bf44153a968857556049df0" is_pre_election: true
I20260926 00:59:42.015796 4785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb81ade8414540ebb01fda61884d1b3c" candidate_uuid: "af0947ff9fcc4c6094ab17942c392fff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9066d18b104344189422cfe810192780" is_pre_election: true
I20260926 00:59:42.015873 4645 raft_consensus.cc:2454] T fb81ade8414540ebb01fda61884d1b3c P 507869ed7bf44153a968857556049df0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af0947ff9fcc4c6094ab17942c392fff in term 0.
I20260926 00:59:42.015904 4785 raft_consensus.cc:2454] T fb81ade8414540ebb01fda61884d1b3c P 9066d18b104344189422cfe810192780 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af0947ff9fcc4c6094ab17942c392fff in term 0.
I20260926 00:59:42.016054 4858 leader_election.cc:302] T fb81ade8414540ebb01fda61884d1b3c P af0947ff9fcc4c6094ab17942c392fff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9066d18b104344189422cfe810192780, af0947ff9fcc4c6094ab17942c392fff; no voters:
I20260926 00:59:42.016160 5011 raft_consensus.cc:2781] T fb81ade8414540ebb01fda61884d1b3c P af0947ff9fcc4c6094ab17942c392fff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 00:59:42.016199 5011 raft_consensus.cc:482] T fb81ade8414540ebb01fda61884d1b3c P af0947ff9fcc4c6094ab17942c392fff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 00:59:42.016227 5011 raft_consensus.cc:3037] T fb81ade8414540ebb01fda61884d1b3c P af0947ff9fcc4c6094ab17942c392fff [term 0 FOLLOWER]: Advancing to term 1
I20260926 00:59:42.016870 5011 raft_consensus.cc:504] T fb81ade8414540ebb01fda61884d1b3c P af0947ff9fcc4c6094ab17942c392fff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507869ed7bf44153a968857556049df0" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37775 } } peers { permanent_uuid: "9066d18b104344189422cfe810192780" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 35513 } } peers { permanent_uuid: "af0947ff9fcc4c6094ab17942c392fff" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 43995 } }
I20260926 00:59:42.016991 5011 leader_election.cc:288] T fb81ade8414540ebb01fda61884d1b3c P af0947ff9fcc4c6094ab17942c392fff [CANDIDATE]: Term 1 election: Requested vote from peers 507869ed7bf44153a968857556049df0 (127.3.75.1:37775), 9066d18b104344189422cfe810192780 (127.3.75.2:35513)
I20260926 00:59:42.017143 4785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb81ade8414540ebb01fda61884d1b3c" candidate_uuid: "af0947ff9fcc4c6094ab17942c392fff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9066d18b104344189422cfe810192780"
I20260926 00:59:42.017127 4645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb81ade8414540ebb01fda61884d1b3c" candidate_uuid: "af0947ff9fcc4c6094ab17942c392fff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "507869ed7bf44153a968857556049df0"
I20260926 00:59:42.017204 4785 raft_consensus.cc:3037] T fb81ade8414540ebb01fda61884d1b3c P 9066d18b104344189422cfe810192780 [term 0 FOLLOWER]: Advancing to term 1
I20260926 00:59:42.017205 4645 raft_consensus.cc:3037] T fb81ade8414540ebb01fda61884d1b3c P 507869ed7bf44153a968857556049df0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 00:59:42.017961 4785 raft_consensus.cc:2454] T fb81ade8414540ebb01fda61884d1b3c P 9066d18b104344189422cfe810192780 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af0947ff9fcc4c6094ab17942c392fff in term 1.
I20260926 00:59:42.017961 4645 raft_consensus.cc:2454] T fb81ade8414540ebb01fda61884d1b3c P 507869ed7bf44153a968857556049df0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af0947ff9fcc4c6094ab17942c392fff in term 1.
I20260926 00:59:42.018119 4858 leader_election.cc:302] T fb81ade8414540ebb01fda61884d1b3c P af0947ff9fcc4c6094ab17942c392fff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9066d18b104344189422cfe810192780, af0947ff9fcc4c6094ab17942c392fff; no voters:
I20260926 00:59:42.018199 5011 raft_consensus.cc:2781] T fb81ade8414540ebb01fda61884d1b3c P af0947ff9fcc4c6094ab17942c392fff [term 1 FOLLOWER]: Leader election won for term 1
I20260926 00:59:42.018275 5011 raft_consensus.cc:686] T fb81ade8414540ebb01fda61884d1b3c P af0947ff9fcc4c6094ab17942c392fff [term 1 LEADER]: Becoming Leader. State: Replica: af0947ff9fcc4c6094ab17942c392fff, State: Running, Role: LEADER
I20260926 00:59:42.018337 5011 consensus_queue.cc:237] T fb81ade8414540ebb01fda61884d1b3c P af0947ff9fcc4c6094ab17942c392fff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507869ed7bf44153a968857556049df0" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37775 } } peers { permanent_uuid: "9066d18b104344189422cfe810192780" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 35513 } } peers { permanent_uuid: "af0947ff9fcc4c6094ab17942c392fff" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 43995 } }
I20260926 00:59:42.018980 4488 catalog_manager.cc:5495] T fb81ade8414540ebb01fda61884d1b3c P af0947ff9fcc4c6094ab17942c392fff reported cstate change: term changed from 0 to 1, leader changed from <none> to af0947ff9fcc4c6094ab17942c392fff (127.3.75.3). New cstate: current_term: 1 leader_uuid: "af0947ff9fcc4c6094ab17942c392fff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507869ed7bf44153a968857556049df0" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9066d18b104344189422cfe810192780" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 35513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af0947ff9fcc4c6094ab17942c392fff" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 43995 } health_report { overall_health: HEALTHY } } }
Sep 26 00:59:42 dist-test-slave-7v10 krb5kdc[4443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384381, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.62@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 00:59:42.064880 5028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:42.055688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34681 (local address 127.0.0.1:47674)
0926 00:59:42.055828 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:42.055838 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:42.055908 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:42.056185 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:42.056186 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:42.056466 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0926 00:59:42.056509 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:42.056511 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:42.057572 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:42.057575 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:42.058543 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:42.058548 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:42.059159 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:42.059160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:42.059219 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:42.059868 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 00:59:42.059874 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 00:59:42.061789 (+ 1915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 00:59:42.063683 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 00:59:42.063691 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 00:59:42.063694 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 00:59:42.063887 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 00:59:42.064111 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 00:59:42.064112 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 00:59:42.064112 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 00:59:42.064170 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 00:59:42.064452 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 00:59:42.064460 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 00:59:42.064505 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 00:59:42.064585 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 00:59:42.064726 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":5,"thread_start_us":82,"threads_started":1}
I20260926 00:59:42.064882 4538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:42.055720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47674 (local address 127.3.75.62:34681)
0926 00:59:42.055768 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:42.055770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:42.055888 (+ 118us) server_negotiation.cc:407] Connection header received
0926 00:59:42.055984 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:42.055984 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:42.056010 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:42.056060 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:42.056605 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:42.057476 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:42.058628 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:42.059017 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:42.059064 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:42.061887 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:42.061888 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:42.061888 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 00:59:42.061895 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:42.063597 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:42.063974 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:42.063975 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:42.063976 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:42.064037 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:42.064237 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:42.064238 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:42.064239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:42.064368 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:42.064448 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:42.064618 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 00:59:42.064785 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":94,"server-negotiator.queue_time_us":23}
I20260926 00:59:42.067206 4488 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:47674:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 00:59:42.067485 4488 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 00:59:42.069031 4765 tablet_service.cc:1467] Processing CreateTablet for tablet 89c2e9df71ea42119ef580d3724a3974 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=50640bb6f25245caa6a76225bf467adb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 00:59:42.069034 4625 tablet_service.cc:1467] Processing CreateTablet for tablet 89c2e9df71ea42119ef580d3724a3974 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=50640bb6f25245caa6a76225bf467adb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 00:59:42.069192 4625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89c2e9df71ea42119ef580d3724a3974. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 00:59:42.069192 4765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89c2e9df71ea42119ef580d3724a3974. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 00:59:42.070542 4995 tablet_bootstrap.cc:492] T 89c2e9df71ea42119ef580d3724a3974 P 507869ed7bf44153a968857556049df0: Bootstrap starting.
I20260926 00:59:42.070693 4905 tablet_service.cc:1467] Processing CreateTablet for tablet 89c2e9df71ea42119ef580d3724a3974 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=50640bb6f25245caa6a76225bf467adb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 00:59:42.070827 4905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89c2e9df71ea42119ef580d3724a3974. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 00:59:42.070987 4995 tablet_bootstrap.cc:654] T 89c2e9df71ea42119ef580d3724a3974 P 507869ed7bf44153a968857556049df0: Neither blocks nor log segments found. Creating new log.
I20260926 00:59:42.072345 4992 tablet_bootstrap.cc:492] T 89c2e9df71ea42119ef580d3724a3974 P af0947ff9fcc4c6094ab17942c392fff: Bootstrap starting.
I20260926 00:59:42.072363 4991 tablet_bootstrap.cc:492] T 89c2e9df71ea42119ef580d3724a3974 P 9066d18b104344189422cfe810192780: Bootstrap starting.
I20260926 00:59:42.072643 4992 tablet_bootstrap.cc:654] T 89c2e9df71ea42119ef580d3724a3974 P af0947ff9fcc4c6094ab17942c392fff: Neither blocks nor log segments found. Creating new log.
I20260926 00:59:42.072754 4991 tablet_bootstrap.cc:654] T 89c2e9df71ea42119ef580d3724a3974 P 9066d18b104344189422cfe810192780: Neither blocks nor log segments found. Creating new log.
I20260926 00:59:42.073179 4992 tablet_bootstrap.cc:492] T 89c2e9df71ea42119ef580d3724a3974 P af0947ff9fcc4c6094ab17942c392fff: No bootstrap required, opened a new log
I20260926 00:59:42.073240 4992 ts_tablet_manager.cc:1397] T 89c2e9df71ea42119ef580d3724a3974 P af0947ff9fcc4c6094ab17942c392fff: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 00:59:42.073349 4992 raft_consensus.cc:348] T 89c2e9df71ea42119ef580d3724a3974 P af0947ff9fcc4c6094ab17942c392fff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507869ed7bf44153a968857556049df0" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37775 } } peers { permanent_uuid: "9066d18b104344189422cfe810192780" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 35513 } } peers { permanent_uuid: "af0947ff9fcc4c6094ab17942c392fff" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 43995 } }
I20260926 00:59:42.073428 4992 raft_consensus.cc:374] T 89c2e9df71ea42119ef580d3724a3974 P af0947ff9fcc4c6094ab17942c392fff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 00:59:42.073455 4992 raft_consensus.cc:729] T 89c2e9df71ea42119ef580d3724a3974 P af0947ff9fcc4c6094ab17942c392fff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af0947ff9fcc4c6094ab17942c392fff, State: Initialized, Role: FOLLOWER
I20260926 00:59:42.073498 4992 consensus_queue.cc:260] T 89c2e9df71ea42119ef580d3724a3974 P af0947ff9fcc4c6094ab17942c392fff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507869ed7bf44153a968857556049df0" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37775 } } peers { permanent_uuid: "9066d18b104344189422cfe810192780" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 35513 } } peers { permanent_uuid: "af0947ff9fcc4c6094ab17942c392fff" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 43995 } }
I20260926 00:59:42.073590 4992 ts_tablet_manager.cc:1428] T 89c2e9df71ea42119ef580d3724a3974 P af0947ff9fcc4c6094ab17942c392fff: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:42.073624 4995 tablet_bootstrap.cc:492] T 89c2e9df71ea42119ef580d3724a3974 P 507869ed7bf44153a968857556049df0: No bootstrap required, opened a new log
I20260926 00:59:42.073678 4995 ts_tablet_manager.cc:1397] T 89c2e9df71ea42119ef580d3724a3974 P 507869ed7bf44153a968857556049df0: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260926 00:59:42.073812 4995 raft_consensus.cc:348] T 89c2e9df71ea42119ef580d3724a3974 P 507869ed7bf44153a968857556049df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507869ed7bf44153a968857556049df0" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37775 } } peers { permanent_uuid: "9066d18b104344189422cfe810192780" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 35513 } } peers { permanent_uuid: "af0947ff9fcc4c6094ab17942c392fff" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 43995 } }
I20260926 00:59:42.073885 4995 raft_consensus.cc:374] T 89c2e9df71ea42119ef580d3724a3974 P 507869ed7bf44153a968857556049df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 00:59:42.073904 4995 raft_consensus.cc:729] T 89c2e9df71ea42119ef580d3724a3974 P 507869ed7bf44153a968857556049df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 507869ed7bf44153a968857556049df0, State: Initialized, Role: FOLLOWER
I20260926 00:59:42.073941 4995 consensus_queue.cc:260] T 89c2e9df71ea42119ef580d3724a3974 P 507869ed7bf44153a968857556049df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507869ed7bf44153a968857556049df0" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37775 } } peers { permanent_uuid: "9066d18b104344189422cfe810192780" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 35513 } } peers { permanent_uuid: "af0947ff9fcc4c6094ab17942c392fff" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 43995 } }
I20260926 00:59:42.074038 4995 ts_tablet_manager.cc:1428] T 89c2e9df71ea42119ef580d3724a3974 P 507869ed7bf44153a968857556049df0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:42.074123 5011 tablet_replica.cc:439] T 89c2e9df71ea42119ef580d3724a3974 P af0947ff9fcc4c6094ab17942c392fff: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507869ed7bf44153a968857556049df0" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37775 } } peers { permanent_uuid: "9066d18b104344189422cfe810192780" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 35513 } } peers { permanent_uuid: "af0947ff9fcc4c6094ab17942c392fff" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 43995 } } }
I20260926 00:59:42.074107 4999 tablet_replica.cc:439] T 89c2e9df71ea42119ef580d3724a3974 P 507869ed7bf44153a968857556049df0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507869ed7bf44153a968857556049df0" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37775 } } peers { permanent_uuid: "9066d18b104344189422cfe810192780" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 35513 } } peers { permanent_uuid: "af0947ff9fcc4c6094ab17942c392fff" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 43995 } } }
I20260926 00:59:42.074194 5011 tablet_replica.cc:442] T 89c2e9df71ea42119ef580d3724a3974 P af0947ff9fcc4c6094ab17942c392fff: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 00:59:42.074218 4999 tablet_replica.cc:442] T 89c2e9df71ea42119ef580d3724a3974 P 507869ed7bf44153a968857556049df0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 00:59:42.074470 4991 tablet_bootstrap.cc:492] T 89c2e9df71ea42119ef580d3724a3974 P 9066d18b104344189422cfe810192780: No bootstrap required, opened a new log
I20260926 00:59:42.074512 4991 ts_tablet_manager.cc:1397] T 89c2e9df71ea42119ef580d3724a3974 P 9066d18b104344189422cfe810192780: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 00:59:42.074638 4991 raft_consensus.cc:348] T 89c2e9df71ea42119ef580d3724a3974 P 9066d18b104344189422cfe810192780 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507869ed7bf44153a968857556049df0" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37775 } } peers { permanent_uuid: "9066d18b104344189422cfe810192780" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 35513 } } peers { permanent_uuid: "af0947ff9fcc4c6094ab17942c392fff" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 43995 } }
I20260926 00:59:42.074724 4991 raft_consensus.cc:374] T 89c2e9df71ea42119ef580d3724a3974 P 9066d18b104344189422cfe810192780 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 00:59:42.074744 4991 raft_consensus.cc:729] T 89c2e9df71ea42119ef580d3724a3974 P 9066d18b104344189422cfe810192780 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9066d18b104344189422cfe810192780, State: Initialized, Role: FOLLOWER
I20260926 00:59:42.074788 4991 consensus_queue.cc:260] T 89c2e9df71ea42119ef580d3724a3974 P 9066d18b104344189422cfe810192780 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507869ed7bf44153a968857556049df0" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37775 } } peers { permanent_uuid: "9066d18b104344189422cfe810192780" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 35513 } } peers { permanent_uuid: "af0947ff9fcc4c6094ab17942c392fff" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 43995 } }
I20260926 00:59:42.074899 4991 ts_tablet_manager.cc:1428] T 89c2e9df71ea42119ef580d3724a3974 P 9066d18b104344189422cfe810192780: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 00:59:42.075111 4996 tablet_replica.cc:439] T 89c2e9df71ea42119ef580d3724a3974 P 9066d18b104344189422cfe810192780: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507869ed7bf44153a968857556049df0" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37775 } } peers { permanent_uuid: "9066d18b104344189422cfe810192780" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 35513 } } peers { permanent_uuid: "af0947ff9fcc4c6094ab17942c392fff" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 43995 } } }
I20260926 00:59:42.075196 4996 tablet_replica.cc:442] T 89c2e9df71ea42119ef580d3724a3974 P 9066d18b104344189422cfe810192780: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 00:59:42.157500 4999 raft_consensus.cc:482] T 89c2e9df71ea42119ef580d3724a3974 P 507869ed7bf44153a968857556049df0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 00:59:42.157608 4999 raft_consensus.cc:504] T 89c2e9df71ea42119ef580d3724a3974 P 507869ed7bf44153a968857556049df0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507869ed7bf44153a968857556049df0" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37775 } } peers { permanent_uuid: "9066d18b104344189422cfe810192780" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 35513 } } peers { permanent_uuid: "af0947ff9fcc4c6094ab17942c392fff" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 43995 } }
I20260926 00:59:42.158022 4999 leader_election.cc:288] T 89c2e9df71ea42119ef580d3724a3974 P 507869ed7bf44153a968857556049df0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9066d18b104344189422cfe810192780 (127.3.75.2:35513), af0947ff9fcc4c6094ab17942c392fff (127.3.75.3:43995)
I20260926 00:59:42.161204 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:42.158287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:34723 (local address 127.3.75.3:43995)
0926 00:59:42.158344 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:42.158346 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:42.158480 (+ 134us) server_negotiation.cc:407] Connection header received
0926 00:59:42.158562 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:42.158563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:42.158579 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:42.158624 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 00:59:42.158809 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:42.159302 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:42.160363 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:42.160807 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:42.160888 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:42.160899 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:42.161066 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 00:59:42.161128 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 00:59:42.161280 4988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:42.158248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:49913 (local address 127.3.75.2:35513)
0926 00:59:42.158297 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:42.158300 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:42.158460 (+ 160us) server_negotiation.cc:407] Connection header received
0926 00:59:42.158564 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:42.158565 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:42.158580 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:42.158632 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 00:59:42.158795 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:42.159336 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:42.160367 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:42.160884 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:42.160949 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:42.160961 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:42.161173 (+ 212us) server_negotiation.cc:299] Negotiation successful
0926 00:59:42.161209 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 00:59:42.161295 5033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:42.158184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.2:35513 (local address 127.3.75.1:49913)
0926 00:59:42.158380 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:42.158395 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:42.158474 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:42.158637 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:42.158637 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:42.158639 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 00:59:42.158724 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:42.158726 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:42.159422 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:42.159423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:42.160265 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:42.160267 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:42.160990 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:42.160990 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:42.161082 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:42.161085 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 00:59:42.161152 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 00:59:42.161191 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":4,"thread_start_us":119,"threads_started":1}
I20260926 00:59:42.161427 5032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:42.158220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.3:43995 (local address 127.3.75.1:34723)
0926 00:59:42.158391 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:42.158408 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:42.158495 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:42.158628 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:42.158629 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:42.158631 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 00:59:42.158734 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:42.158736 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:42.159393 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:42.159395 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:42.160263 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:42.160265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:42.160888 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:42.160890 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:42.160968 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:42.160973 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 00:59:42.161299 (+ 326us) client_negotiation.cc:240] Negotiation successful
0926 00:59:42.161339 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":41,"thread_start_us":102,"threads_started":1}
I20260926 00:59:42.161592 4785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89c2e9df71ea42119ef580d3724a3974" candidate_uuid: "507869ed7bf44153a968857556049df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9066d18b104344189422cfe810192780" is_pre_election: true
I20260926 00:59:42.161685 4785 raft_consensus.cc:2454] T 89c2e9df71ea42119ef580d3724a3974 P 9066d18b104344189422cfe810192780 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 507869ed7bf44153a968857556049df0 in term 0.
I20260926 00:59:42.161739 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89c2e9df71ea42119ef580d3724a3974" candidate_uuid: "507869ed7bf44153a968857556049df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af0947ff9fcc4c6094ab17942c392fff" is_pre_election: true
I20260926 00:59:42.161854 4925 raft_consensus.cc:2454] T 89c2e9df71ea42119ef580d3724a3974 P af0947ff9fcc4c6094ab17942c392fff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 507869ed7bf44153a968857556049df0 in term 0.
I20260926 00:59:42.161873 4578 leader_election.cc:302] T 89c2e9df71ea42119ef580d3724a3974 P 507869ed7bf44153a968857556049df0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 507869ed7bf44153a968857556049df0, 9066d18b104344189422cfe810192780; no voters:
I20260926 00:59:42.162016 4999 raft_consensus.cc:2781] T 89c2e9df71ea42119ef580d3724a3974 P 507869ed7bf44153a968857556049df0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 00:59:42.162065 4999 raft_consensus.cc:482] T 89c2e9df71ea42119ef580d3724a3974 P 507869ed7bf44153a968857556049df0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 00:59:42.162093 4999 raft_consensus.cc:3037] T 89c2e9df71ea42119ef580d3724a3974 P 507869ed7bf44153a968857556049df0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 00:59:42.162756 4999 raft_consensus.cc:504] T 89c2e9df71ea42119ef580d3724a3974 P 507869ed7bf44153a968857556049df0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507869ed7bf44153a968857556049df0" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37775 } } peers { permanent_uuid: "9066d18b104344189422cfe810192780" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 35513 } } peers { permanent_uuid: "af0947ff9fcc4c6094ab17942c392fff" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 43995 } }
I20260926 00:59:42.162933 4999 leader_election.cc:288] T 89c2e9df71ea42119ef580d3724a3974 P 507869ed7bf44153a968857556049df0 [CANDIDATE]: Term 1 election: Requested vote from peers 9066d18b104344189422cfe810192780 (127.3.75.2:35513), af0947ff9fcc4c6094ab17942c392fff (127.3.75.3:43995)
I20260926 00:59:42.163087 4785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89c2e9df71ea42119ef580d3724a3974" candidate_uuid: "507869ed7bf44153a968857556049df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9066d18b104344189422cfe810192780"
I20260926 00:59:42.163164 4785 raft_consensus.cc:3037] T 89c2e9df71ea42119ef580d3724a3974 P 9066d18b104344189422cfe810192780 [term 0 FOLLOWER]: Advancing to term 1
I20260926 00:59:42.163759 4785 raft_consensus.cc:2454] T 89c2e9df71ea42119ef580d3724a3974 P 9066d18b104344189422cfe810192780 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 507869ed7bf44153a968857556049df0 in term 1.
I20260926 00:59:42.163880 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89c2e9df71ea42119ef580d3724a3974" candidate_uuid: "507869ed7bf44153a968857556049df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af0947ff9fcc4c6094ab17942c392fff"
I20260926 00:59:42.163929 4578 leader_election.cc:302] T 89c2e9df71ea42119ef580d3724a3974 P 507869ed7bf44153a968857556049df0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 507869ed7bf44153a968857556049df0, 9066d18b104344189422cfe810192780; no voters:
I20260926 00:59:42.163959 4925 raft_consensus.cc:3037] T 89c2e9df71ea42119ef580d3724a3974 P af0947ff9fcc4c6094ab17942c392fff [term 0 FOLLOWER]: Advancing to term 1
I20260926 00:59:42.164021 4999 raft_consensus.cc:2781] T 89c2e9df71ea42119ef580d3724a3974 P 507869ed7bf44153a968857556049df0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 00:59:42.164183 4999 raft_consensus.cc:686] T 89c2e9df71ea42119ef580d3724a3974 P 507869ed7bf44153a968857556049df0 [term 1 LEADER]: Becoming Leader. State: Replica: 507869ed7bf44153a968857556049df0, State: Running, Role: LEADER
I20260926 00:59:42.164259 4999 consensus_queue.cc:237] T 89c2e9df71ea42119ef580d3724a3974 P 507869ed7bf44153a968857556049df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507869ed7bf44153a968857556049df0" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37775 } } peers { permanent_uuid: "9066d18b104344189422cfe810192780" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 35513 } } peers { permanent_uuid: "af0947ff9fcc4c6094ab17942c392fff" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 43995 } }
I20260926 00:59:42.164620 5034 tablet_replica.cc:439] T 89c2e9df71ea42119ef580d3724a3974 P 507869ed7bf44153a968857556049df0: TxnStatusTablet state changed. Reason: New leader 507869ed7bf44153a968857556049df0. Latest consensus state: current_term: 1 leader_uuid: "507869ed7bf44153a968857556049df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507869ed7bf44153a968857556049df0" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37775 } } peers { permanent_uuid: "9066d18b104344189422cfe810192780" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 35513 } } peers { permanent_uuid: "af0947ff9fcc4c6094ab17942c392fff" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 43995 } } }
I20260926 00:59:42.164794 5034 tablet_replica.cc:442] T 89c2e9df71ea42119ef580d3724a3974 P 507869ed7bf44153a968857556049df0: This TxnStatusTablet replica's current role is: LEADER
I20260926 00:59:42.164803 4925 raft_consensus.cc:2454] T 89c2e9df71ea42119ef580d3724a3974 P af0947ff9fcc4c6094ab17942c392fff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 507869ed7bf44153a968857556049df0 in term 1.
I20260926 00:59:42.164961 4487 catalog_manager.cc:5495] T 89c2e9df71ea42119ef580d3724a3974 P 507869ed7bf44153a968857556049df0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 507869ed7bf44153a968857556049df0 (127.3.75.1). New cstate: current_term: 1 leader_uuid: "507869ed7bf44153a968857556049df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507869ed7bf44153a968857556049df0" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9066d18b104344189422cfe810192780" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 35513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af0947ff9fcc4c6094ab17942c392fff" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 43995 } health_report { overall_health: UNKNOWN } } }
I20260926 00:59:42.165037 5036 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 00:59:42.165134 5036 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 00:59:42.189355 5028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:42.186275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.1:37775 (local address 127.0.0.1:40016)
0926 00:59:42.186359 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:42.186371 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:42.186416 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:42.186748 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:42.186748 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:42.186749 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 00:59:42.186797 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:42.186799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:42.187707 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:42.187708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:42.188339 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:42.188341 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:42.188823 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:42.188824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:42.188874 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:42.188888 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 00:59:42.189169 (+ 281us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 00:59:42.189178 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 00:59:42.189233 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 00:59:42.189277 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 00:59:42.189368 5002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:42.186506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40016 (local address 127.3.75.1:37775)
0926 00:59:42.186578 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:42.186581 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:42.186591 (+ 10us) server_negotiation.cc:407] Connection header received
0926 00:59:42.186631 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:42.186632 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:42.186655 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:42.186735 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 00:59:42.186864 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:42.187630 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:42.188411 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:42.188733 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:42.188807 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:42.188989 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 00:59:42.189098 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 00:59:42.189185 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:42.189238 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 00:59:42.189260 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 00:59:42.190635 4785 raft_consensus.cc:1261] T 89c2e9df71ea42119ef580d3724a3974 P 9066d18b104344189422cfe810192780 [term 1 FOLLOWER]: Refusing update from remote peer 507869ed7bf44153a968857556049df0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 00:59:42.190693 4925 raft_consensus.cc:1261] T 89c2e9df71ea42119ef580d3724a3974 P af0947ff9fcc4c6094ab17942c392fff [term 1 FOLLOWER]: Refusing update from remote peer 507869ed7bf44153a968857556049df0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 00:59:42.190951 5034 consensus_queue.cc:1035] T 89c2e9df71ea42119ef580d3724a3974 P 507869ed7bf44153a968857556049df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9066d18b104344189422cfe810192780" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 35513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 00:59:42.191102 4999 consensus_queue.cc:1035] T 89c2e9df71ea42119ef580d3724a3974 P 507869ed7bf44153a968857556049df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0947ff9fcc4c6094ab17942c392fff" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 43995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 00:59:42.191640 4996 tablet_replica.cc:439] T 89c2e9df71ea42119ef580d3724a3974 P 9066d18b104344189422cfe810192780: TxnStatusTablet state changed. Reason: New leader 507869ed7bf44153a968857556049df0. Latest consensus state: current_term: 1 leader_uuid: "507869ed7bf44153a968857556049df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507869ed7bf44153a968857556049df0" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37775 } } peers { permanent_uuid: "9066d18b104344189422cfe810192780" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 35513 } } peers { permanent_uuid: "af0947ff9fcc4c6094ab17942c392fff" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 43995 } } }
I20260926 00:59:42.191723 4996 tablet_replica.cc:442] T 89c2e9df71ea42119ef580d3724a3974 P 9066d18b104344189422cfe810192780: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 00:59:42.191880 5011 tablet_replica.cc:439] T 89c2e9df71ea42119ef580d3724a3974 P af0947ff9fcc4c6094ab17942c392fff: TxnStatusTablet state changed. Reason: New leader 507869ed7bf44153a968857556049df0. Latest consensus state: current_term: 1 leader_uuid: "507869ed7bf44153a968857556049df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507869ed7bf44153a968857556049df0" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37775 } } peers { permanent_uuid: "9066d18b104344189422cfe810192780" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 35513 } } peers { permanent_uuid: "af0947ff9fcc4c6094ab17942c392fff" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 43995 } } }
I20260926 00:59:42.191984 5011 tablet_replica.cc:442] T 89c2e9df71ea42119ef580d3724a3974 P af0947ff9fcc4c6094ab17942c392fff: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 00:59:42.192638 5034 tablet_replica.cc:439] T 89c2e9df71ea42119ef580d3724a3974 P 507869ed7bf44153a968857556049df0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "507869ed7bf44153a968857556049df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507869ed7bf44153a968857556049df0" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37775 } } peers { permanent_uuid: "9066d18b104344189422cfe810192780" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 35513 } } peers { permanent_uuid: "af0947ff9fcc4c6094ab17942c392fff" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 43995 } } }
I20260926 00:59:42.192723 5034 tablet_replica.cc:442] T 89c2e9df71ea42119ef580d3724a3974 P 507869ed7bf44153a968857556049df0: This TxnStatusTablet replica's current role is: LEADER
I20260926 00:59:42.192925 5034 tablet_replica.cc:439] T 89c2e9df71ea42119ef580d3724a3974 P 507869ed7bf44153a968857556049df0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "507869ed7bf44153a968857556049df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507869ed7bf44153a968857556049df0" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37775 } } peers { permanent_uuid: "9066d18b104344189422cfe810192780" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 35513 } } peers { permanent_uuid: "af0947ff9fcc4c6094ab17942c392fff" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 43995 } } }
I20260926 00:59:42.193017 5034 tablet_replica.cc:442] T 89c2e9df71ea42119ef580d3724a3974 P 507869ed7bf44153a968857556049df0: This TxnStatusTablet replica's current role is: LEADER
I20260926 00:59:42.193397 5011 tablet_replica.cc:439] T 89c2e9df71ea42119ef580d3724a3974 P af0947ff9fcc4c6094ab17942c392fff: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "507869ed7bf44153a968857556049df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507869ed7bf44153a968857556049df0" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37775 } } peers { permanent_uuid: "9066d18b104344189422cfe810192780" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 35513 } } peers { permanent_uuid: "af0947ff9fcc4c6094ab17942c392fff" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 43995 } } }
I20260926 00:59:42.193473 5011 tablet_replica.cc:442] T 89c2e9df71ea42119ef580d3724a3974 P af0947ff9fcc4c6094ab17942c392fff: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 00:59:42.193538 4996 tablet_replica.cc:439] T 89c2e9df71ea42119ef580d3724a3974 P 9066d18b104344189422cfe810192780: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "507869ed7bf44153a968857556049df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507869ed7bf44153a968857556049df0" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37775 } } peers { permanent_uuid: "9066d18b104344189422cfe810192780" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 35513 } } peers { permanent_uuid: "af0947ff9fcc4c6094ab17942c392fff" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 43995 } } }
I20260926 00:59:42.193620 4996 tablet_replica.cc:442] T 89c2e9df71ea42119ef580d3724a3974 P 9066d18b104344189422cfe810192780: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 00:59:42.198939 4977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:42.196675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.1:37775 (local address 127.3.75.3:56243)
0926 00:59:42.196745 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:42.196757 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:42.196804 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:42.196919 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:42.196919 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:42.196920 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 00:59:42.196964 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:42.196965 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:42.197556 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:42.197557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:42.198112 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:42.198114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:42.198478 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:42.198480 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:42.198557 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:42.198569 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 00:59:42.198758 (+ 189us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 00:59:42.198760 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 00:59:42.198812 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 00:59:42.198853 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 00:59:42.198931 5002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:42.196744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.3:56243 (local address 127.3.75.1:37775)
0926 00:59:42.196781 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:42.196783 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:42.196790 (+ 7us) server_negotiation.cc:407] Connection header received
0926 00:59:42.196853 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:42.196854 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:42.196864 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:42.196916 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 00:59:42.197031 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:42.197469 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:42.198204 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:42.198392 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:42.198472 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:42.198660 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 00:59:42.198697 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 00:59:42.198758 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:42.198817 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 00:59:42.198836 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 00:59:42.201179 4645 raft_consensus.cc:1261] T fb81ade8414540ebb01fda61884d1b3c P 507869ed7bf44153a968857556049df0 [term 1 FOLLOWER]: Refusing update from remote peer af0947ff9fcc4c6094ab17942c392fff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 00:59:42.201220 4785 raft_consensus.cc:1261] T fb81ade8414540ebb01fda61884d1b3c P 9066d18b104344189422cfe810192780 [term 1 FOLLOWER]: Refusing update from remote peer af0947ff9fcc4c6094ab17942c392fff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 00:59:42.201388 5003 consensus_queue.cc:1035] T fb81ade8414540ebb01fda61884d1b3c P af0947ff9fcc4c6094ab17942c392fff [LEADER]: Connected to new peer: Peer: permanent_uuid: "9066d18b104344189422cfe810192780" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 35513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 00:59:42.201561 5003 consensus_queue.cc:1035] T fb81ade8414540ebb01fda61884d1b3c P af0947ff9fcc4c6094ab17942c392fff [LEADER]: Connected to new peer: Peer: permanent_uuid: "507869ed7bf44153a968857556049df0" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 00:59:42.210120 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:42.207105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:52059 (local address 127.3.75.3:43995)
0926 00:59:42.207142 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:42.207144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:42.207307 (+ 163us) server_negotiation.cc:407] Connection header received
0926 00:59:42.207335 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:42.207336 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:42.207346 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:42.207414 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 00:59:42.207539 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:42.208259 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:42.209385 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:42.209699 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:42.209771 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:42.209907 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 00:59:42.209940 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 00:59:42.209993 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:42.210032 (+ 39us) server_negotiation.cc:299] Negotiation successful
0926 00:59:42.210052 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 00:59:42.210137 5046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:42.207080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.3:43995 (local address 127.3.75.1:52059)
0926 00:59:42.207236 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:42.207245 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:42.207293 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:42.207414 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:42.207415 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:42.207423 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 00:59:42.207474 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:42.207475 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:42.208354 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:42.208354 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:42.209312 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:42.209314 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:42.209774 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:42.209774 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:42.209821 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:42.209834 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 00:59:42.210000 (+ 166us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 00:59:42.210002 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 00:59:42.210029 (+ 27us) client_negotiation.cc:240] Negotiation successful
0926 00:59:42.210069 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":86,"threads_started":1}
I20260926 00:59:42.218480 4488 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 00:59:42.218603 4488 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 00:59:42.219308 4488 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 03a7c52ba87c4b0a875dec905ad499a1: Sending DeleteTablet for 3 replicas of tablet fb81ade8414540ebb01fda61884d1b3c
I20260926 00:59:42.219627 4905 tablet_service.cc:1514] Processing DeleteTablet for tablet fb81ade8414540ebb01fda61884d1b3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 00:59:42 UTC) from {username='kudu', principal='kudu/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:59224
I20260926 00:59:42.219627 4765 tablet_service.cc:1514] Processing DeleteTablet for tablet fb81ade8414540ebb01fda61884d1b3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 00:59:42 UTC) from {username='kudu', principal='kudu/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:35202
I20260926 00:59:42.219635 4625 tablet_service.cc:1514] Processing DeleteTablet for tablet fb81ade8414540ebb01fda61884d1b3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 00:59:42 UTC) from {username='kudu', principal='kudu/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:40012
I20260926 00:59:42.219724 5016 tablet_replica.cc:330] T fb81ade8414540ebb01fda61884d1b3c P 9066d18b104344189422cfe810192780: stopping tablet replica
I20260926 00:59:42.219733 5018 tablet_replica.cc:330] T fb81ade8414540ebb01fda61884d1b3c P 507869ed7bf44153a968857556049df0: stopping tablet replica
I20260926 00:59:42.219731 5017 tablet_replica.cc:330] T fb81ade8414540ebb01fda61884d1b3c P af0947ff9fcc4c6094ab17942c392fff: stopping tablet replica
I20260926 00:59:42.219772 5016 raft_consensus.cc:2229] T fb81ade8414540ebb01fda61884d1b3c P 9066d18b104344189422cfe810192780 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 00:59:42.219782 5018 raft_consensus.cc:2229] T fb81ade8414540ebb01fda61884d1b3c P 507869ed7bf44153a968857556049df0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 00:59:42.219790 5017 raft_consensus.cc:2229] T fb81ade8414540ebb01fda61884d1b3c P af0947ff9fcc4c6094ab17942c392fff [term 1 LEADER]: Raft consensus shutting down.
I20260926 00:59:42.219820 5016 raft_consensus.cc:2258] T fb81ade8414540ebb01fda61884d1b3c P 9066d18b104344189422cfe810192780 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 00:59:42.219828 5018 raft_consensus.cc:2258] T fb81ade8414540ebb01fda61884d1b3c P 507869ed7bf44153a968857556049df0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 00:59:42.219844 5017 raft_consensus.cc:2258] T fb81ade8414540ebb01fda61884d1b3c P af0947ff9fcc4c6094ab17942c392fff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 00:59:42.220211 5016 ts_tablet_manager.cc:1905] T fb81ade8414540ebb01fda61884d1b3c P 9066d18b104344189422cfe810192780: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 00:59:42.220216 5017 ts_tablet_manager.cc:1905] T fb81ade8414540ebb01fda61884d1b3c P af0947ff9fcc4c6094ab17942c392fff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 00:59:42.220232 5018 ts_tablet_manager.cc:1905] T fb81ade8414540ebb01fda61884d1b3c P 507869ed7bf44153a968857556049df0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 00:59:42.222137 5018 ts_tablet_manager.cc:1918] T fb81ade8414540ebb01fda61884d1b3c P 507869ed7bf44153a968857556049df0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 00:59:42.222143 5017 ts_tablet_manager.cc:1918] T fb81ade8414540ebb01fda61884d1b3c P af0947ff9fcc4c6094ab17942c392fff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 00:59:42.222169 5016 ts_tablet_manager.cc:1918] T fb81ade8414540ebb01fda61884d1b3c P 9066d18b104344189422cfe810192780: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 00:59:42.222198 5017 log.cc:1196] T fb81ade8414540ebb01fda61884d1b3c P af0947ff9fcc4c6094ab17942c392fff: Deleting WAL directory at /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/ts-2/wal/wals/fb81ade8414540ebb01fda61884d1b3c
I20260926 00:59:42.222198 5018 log.cc:1196] T fb81ade8414540ebb01fda61884d1b3c P 507869ed7bf44153a968857556049df0: Deleting WAL directory at /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/ts-0/wal/wals/fb81ade8414540ebb01fda61884d1b3c
I20260926 00:59:42.222223 5016 log.cc:1196] T fb81ade8414540ebb01fda61884d1b3c P 9066d18b104344189422cfe810192780: Deleting WAL directory at /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384372073861-3372-0/minicluster-data/ts-1/wal/wals/fb81ade8414540ebb01fda61884d1b3c
I20260926 00:59:42.222435 5018 ts_tablet_manager.cc:1939] T fb81ade8414540ebb01fda61884d1b3c P 507869ed7bf44153a968857556049df0: Deleting consensus metadata
I20260926 00:59:42.222435 5017 ts_tablet_manager.cc:1939] T fb81ade8414540ebb01fda61884d1b3c P af0947ff9fcc4c6094ab17942c392fff: Deleting consensus metadata
I20260926 00:59:42.222447 5016 ts_tablet_manager.cc:1939] T fb81ade8414540ebb01fda61884d1b3c P 9066d18b104344189422cfe810192780: Deleting consensus metadata
I20260926 00:59:42.222724 4472 catalog_manager.cc:4841] TS af0947ff9fcc4c6094ab17942c392fff (127.3.75.3:43995): tablet fb81ade8414540ebb01fda61884d1b3c (table test-table [id=21f5e2e32b204f0bbf74486af94df07f]) successfully deleted
I20260926 00:59:42.222755 4474 catalog_manager.cc:4841] TS 9066d18b104344189422cfe810192780 (127.3.75.2:35513): tablet fb81ade8414540ebb01fda61884d1b3c (table test-table [id=21f5e2e32b204f0bbf74486af94df07f]) successfully deleted
I20260926 00:59:42.222842 4475 catalog_manager.cc:4841] TS 507869ed7bf44153a968857556049df0 (127.3.75.1:37775): tablet fb81ade8414540ebb01fda61884d1b3c (table test-table [id=21f5e2e32b204f0bbf74486af94df07f]) successfully deleted
Sep 26 00:59:42 dist-test-slave-7v10 krb5kdc[4443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384381, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.75.1@KRBTEST.COM
I20260926 00:59:42.231948 5002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:42.223124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40018 (local address 127.3.75.1:37775)
0926 00:59:42.223161 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:42.223163 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:42.223304 (+ 141us) server_negotiation.cc:407] Connection header received
0926 00:59:42.223381 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:42.223382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:42.223393 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:42.223442 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:42.224099 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:42.224642 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:42.225611 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:42.225803 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:42.225891 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:42.229033 (+ 3142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:42.229042 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:42.229043 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 00:59:42.229051 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:42.230694 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:42.231149 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:42.231152 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:42.231153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:42.231211 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:42.231412 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:42.231413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:42.231414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:42.231553 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:42.231714 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:42.231757 (+ 43us) server_negotiation.cc:299] Negotiation successful
0926 00:59:42.231839 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260926 00:59:42.232193 4685 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40018
I20260926 00:59:42.239684 4538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:42.233096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47688 (local address 127.3.75.62:34681)
0926 00:59:42.233140 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:42.233142 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:42.233178 (+ 36us) server_negotiation.cc:407] Connection header received
0926 00:59:42.233242 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:42.233243 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:42.233252 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:42.233317 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:42.233767 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:42.234577 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:42.235433 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:42.235723 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:42.235793 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:42.237148 (+ 1355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:42.237150 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:42.237150 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 00:59:42.237156 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:42.238518 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:42.238938 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:42.238939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:42.238940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:42.239022 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:42.239239 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:42.239240 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:42.239240 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:42.239341 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:42.239414 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:42.239568 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 00:59:42.239624 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 00:59:42.240098 4488 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47688
I20260926 00:59:42.240849 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 4560
I20260926 00:59:42.246543 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 4700
I20260926 00:59:42.250952 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 4840
I20260926 00:59:42.255935 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 4456
2026-09-26T00:59:42Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5932 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 00:59:42 dist-test-slave-7v10 krb5kdc[5054](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 00:59:42 dist-test-slave-7v10 krb5kdc[5054](info): set up 2 sockets
Sep 26 00:59:42 dist-test-slave-7v10 krb5kdc[5054](info): commencing operation
krb5kdc: starting...
W20260926 00:59:43.285754 3372 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Sep 26 00:59:43 dist-test-slave-7v10 krb5kdc[5054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T00:59:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T00:59:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.75.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab.
Entry for principal kudu/127.3.75.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab.
I20260926 00:59:43.398104 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:33135
--webserver_interface=127.3.75.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:37213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.75.62:33135
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 00:59:43.476521 5067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 00:59:43.476747 5067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 00:59:43.476778 5067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 00:59:43.477886 5067 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 00:59:43.477947 5067 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 00:59:43.477970 5067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 00:59:43.478003 5067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 00:59:43.478032 5067 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 00:59:43.478061 5067 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 00:59:43.479594 5067 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:37213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.75.62:33135
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:33135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 00:59:43.479866 5067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 00:59:43.480080 5067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 00:59:43.484740 5073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:43.484751 5075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:43.484747 5072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 00:59:43.484987 5067 server_base.cc:1034] running on GCE node
I20260926 00:59:43.485217 5067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 00:59:43.485499 5067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 00:59:43.486656 5067 hybrid_clock.cc:648] HybridClock initialized: now 1790384383486645 us; error 44 us; skew 500 ppm
Sep 26 00:59:43 dist-test-slave-7v10 krb5kdc[5054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384383, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 00:59:43.489233 5067 init.cc:377] Logged in from keytab as kudu/127.3.75.62@KRBTEST.COM (short username kudu)
I20260926 00:59:43.489727 5067 webserver.cc:460] Webserver started at http://127.3.75.62:45101/ using document root <none> and password file <none>
I20260926 00:59:43.489972 5067 fs_manager.cc:359] Metadata directory not provided
I20260926 00:59:43.490041 5067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 00:59:43.490190 5067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 00:59:43.491317 5067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/master-0/data/instance:
uuid: "b5d54f69067e4c24a51d89252d078c71"
format_stamp: "Formatted at 2026-09-26 00:59:43 on dist-test-slave-7v10"
server_key: "86a402a01ec0f56179d62b0e6f9214ef"
server_key_iv: "36ed397b16a46234259cd80b1a3063bf"
server_key_version: "encryptionkey@0"
I20260926 00:59:43.491715 5067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/master-0/wal/instance:
uuid: "b5d54f69067e4c24a51d89252d078c71"
format_stamp: "Formatted at 2026-09-26 00:59:43 on dist-test-slave-7v10"
server_key: "86a402a01ec0f56179d62b0e6f9214ef"
server_key_iv: "36ed397b16a46234259cd80b1a3063bf"
server_key_version: "encryptionkey@0"
I20260926 00:59:43.493067 5067 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 00:59:43.493804 5082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:43.493975 5067 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 00:59:43.494083 5067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/master-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/master-0/wal
uuid: "b5d54f69067e4c24a51d89252d078c71"
format_stamp: "Formatted at 2026-09-26 00:59:43 on dist-test-slave-7v10"
server_key: "86a402a01ec0f56179d62b0e6f9214ef"
server_key_iv: "36ed397b16a46234259cd80b1a3063bf"
server_key_version: "encryptionkey@0"
I20260926 00:59:43.494158 5067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 00:59:43.512974 5067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 00:59:43.515349 5067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 00:59:43.515518 5067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 00:59:43.520164 5067 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.62:33135
I20260926 00:59:43.520215 5144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.62:33135 every 8 connection(s)
I20260926 00:59:43.520543 5067 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
I20260926 00:59:43.521198 5145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 00:59:43.522360 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 5067
I20260926 00:59:43.522462 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/master-0/wal/instance
I20260926 00:59:43.522658 3372 external_mini_cluster.cc:1428] Setting key ac8e288a34eadf4b53fc012445b83ec5
I20260926 00:59:43.523940 5145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5d54f69067e4c24a51d89252d078c71: Bootstrap starting.
I20260926 00:59:43.524545 5145 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5d54f69067e4c24a51d89252d078c71: Neither blocks nor log segments found. Creating new log.
I20260926 00:59:43.524897 5145 log.cc:824] T 00000000000000000000000000000000 P b5d54f69067e4c24a51d89252d078c71: Log is configured to *not* fsync() on all Append() calls
I20260926 00:59:43.525636 5145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5d54f69067e4c24a51d89252d078c71: No bootstrap required, opened a new log
I20260926 00:59:43.527885 5145 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5d54f69067e4c24a51d89252d078c71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5d54f69067e4c24a51d89252d078c71" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 33135 } }
I20260926 00:59:43.528024 5145 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5d54f69067e4c24a51d89252d078c71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 00:59:43.528047 5145 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5d54f69067e4c24a51d89252d078c71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5d54f69067e4c24a51d89252d078c71, State: Initialized, Role: FOLLOWER
I20260926 00:59:43.528184 5145 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5d54f69067e4c24a51d89252d078c71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5d54f69067e4c24a51d89252d078c71" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 33135 } }
I20260926 00:59:43.528254 5145 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5d54f69067e4c24a51d89252d078c71 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 00:59:43.528276 5145 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5d54f69067e4c24a51d89252d078c71 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 00:59:43.528342 5145 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5d54f69067e4c24a51d89252d078c71 [term 0 FOLLOWER]: Advancing to term 1
I20260926 00:59:43.528837 5145 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5d54f69067e4c24a51d89252d078c71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5d54f69067e4c24a51d89252d078c71" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 33135 } }
I20260926 00:59:43.528919 5145 leader_election.cc:302] T 00000000000000000000000000000000 P b5d54f69067e4c24a51d89252d078c71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5d54f69067e4c24a51d89252d078c71; no voters:
I20260926 00:59:43.529102 5145 leader_election.cc:288] T 00000000000000000000000000000000 P b5d54f69067e4c24a51d89252d078c71 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 00:59:43.529158 5150 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5d54f69067e4c24a51d89252d078c71 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 00:59:43.529311 5150 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5d54f69067e4c24a51d89252d078c71 [term 1 LEADER]: Becoming Leader. State: Replica: b5d54f69067e4c24a51d89252d078c71, State: Running, Role: LEADER
I20260926 00:59:43.529379 5145 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5d54f69067e4c24a51d89252d078c71 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 00:59:43.529407 5150 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5d54f69067e4c24a51d89252d078c71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5d54f69067e4c24a51d89252d078c71" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 33135 } }
Sep 26 00:59:43 dist-test-slave-7v10 krb5kdc[5054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 00:59:43.529716 5150 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5d54f69067e4c24a51d89252d078c71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5d54f69067e4c24a51d89252d078c71. Latest consensus state: current_term: 1 leader_uuid: "b5d54f69067e4c24a51d89252d078c71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5d54f69067e4c24a51d89252d078c71" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 33135 } } }
I20260926 00:59:43.529798 5150 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5d54f69067e4c24a51d89252d078c71 [sys.catalog]: This master's current role is: LEADER
I20260926 00:59:43.529724 5152 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5d54f69067e4c24a51d89252d078c71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5d54f69067e4c24a51d89252d078c71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5d54f69067e4c24a51d89252d078c71" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 33135 } } }
I20260926 00:59:43.529983 5152 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5d54f69067e4c24a51d89252d078c71 [sys.catalog]: This master's current role is: LEADER
W20260926 00:59:43.531113 5166 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b5d54f69067e4c24a51d89252d078c71: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 00:59:43.531190 5166 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 00:59:43.531409 5156 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 00:59:43.531601 5156 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 00:59:43.532918 5156 catalog_manager.cc:1347] Generated new cluster ID: 4ff5798afb4d42ffbb7e0f856b4eade6
I20260926 00:59:43.532965 5156 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 00:59:43.534392 5148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:43.523219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58206 (local address 127.3.75.62:33135)
0926 00:59:43.523401 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:43.523416 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:43.523439 (+ 23us) server_negotiation.cc:407] Connection header received
0926 00:59:43.523836 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:43.523843 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:43.523888 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:43.524062 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:43.524735 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:43.526065 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:43.526786 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:43.527414 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:43.527555 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:43.530214 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:43.530219 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:43.530219 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 00:59:43.530230 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:43.532307 (+ 2077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:43.532674 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:43.532676 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:43.532677 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:43.532784 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:43.533084 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:43.533085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:43.533086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:43.533381 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:43.533519 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:43.534002 (+ 483us) server_negotiation.cc:299] Negotiation successful
0926 00:59:43.534167 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":87,"threads_started":1}
I20260926 00:59:43.544500 5156 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 00:59:43.544973 5156 catalog_manager.cc:1504] Loading token signing keys...
I20260926 00:59:43.548115 5156 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5d54f69067e4c24a51d89252d078c71: Generated new TSK 0
I20260926 00:59:43.548287 5156 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.75.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab.
Entry for principal kudu/127.3.75.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab.
I20260926 00:59:43.575628 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.1:0
--local_ip_for_outbound_sockets=127.3.75.1
--webserver_interface=127.3.75.1
--webserver_port=0
--tserver_master_addrs=127.3.75.62:33135
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab
--principal=kudu/127.3.75.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:37213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 00:59:43.666111 5171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 00:59:43.666343 5171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 00:59:43.666383 5171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 00:59:43.667634 5171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 00:59:43.667713 5171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 00:59:43.667752 5171 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 00:59:43.667796 5171 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 00:59:43.667836 5171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.1
I20260926 00:59:43.669265 5171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:37213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab
--principal=kudu/127.3.75.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.75.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.75.62:33135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.1
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 00:59:43.669528 5171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 00:59:43.669755 5171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 00:59:43.674746 5179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:43.674762 5177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:43.675092 5176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 00:59:43.675460 5171 server_base.cc:1034] running on GCE node
I20260926 00:59:43.675701 5171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 00:59:43.675952 5171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 00:59:43.677099 5171 hybrid_clock.cc:648] HybridClock initialized: now 1790384383677085 us; error 31 us; skew 500 ppm
Sep 26 00:59:43 dist-test-slave-7v10 krb5kdc[5054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384383, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 00:59:43.679589 5171 init.cc:377] Logged in from keytab as kudu/127.3.75.1@KRBTEST.COM (short username kudu)
I20260926 00:59:43.680029 5171 webserver.cc:460] Webserver started at http://127.3.75.1:34133/ using document root <none> and password file <none>
I20260926 00:59:43.680220 5171 fs_manager.cc:359] Metadata directory not provided
I20260926 00:59:43.680259 5171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 00:59:43.680346 5171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 00:59:43.681345 5171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/ts-0/data/instance:
uuid: "5ea28b33245c4862a7c4dfd2230860f7"
format_stamp: "Formatted at 2026-09-26 00:59:43 on dist-test-slave-7v10"
server_key: "6fd30f09ff529cb5212df012116d65ea"
server_key_iv: "991fddeb12812126d55b336d917b8dc2"
server_key_version: "encryptionkey@0"
I20260926 00:59:43.681644 5171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/ts-0/wal/instance:
uuid: "5ea28b33245c4862a7c4dfd2230860f7"
format_stamp: "Formatted at 2026-09-26 00:59:43 on dist-test-slave-7v10"
server_key: "6fd30f09ff529cb5212df012116d65ea"
server_key_iv: "991fddeb12812126d55b336d917b8dc2"
server_key_version: "encryptionkey@0"
I20260926 00:59:43.682865 5171 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 00:59:43.683606 5186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:43.683772 5171 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 00:59:43.683856 5171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/ts-0/wal
uuid: "5ea28b33245c4862a7c4dfd2230860f7"
format_stamp: "Formatted at 2026-09-26 00:59:43 on dist-test-slave-7v10"
server_key: "6fd30f09ff529cb5212df012116d65ea"
server_key_iv: "991fddeb12812126d55b336d917b8dc2"
server_key_version: "encryptionkey@0"
I20260926 00:59:43.683909 5171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 00:59:43.691309 5171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 00:59:43.693542 5171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 00:59:43.693653 5171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 00:59:43.694082 5171 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 00:59:43.694121 5171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:43.694144 5171 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 00:59:43.694157 5171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:43.700982 5171 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.1:40211
I20260926 00:59:43.701035 5301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.1:40211 every 8 connection(s)
I20260926 00:59:43.701341 5171 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
Sep 26 00:59:43 dist-test-slave-7v10 krb5kdc[5054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384383, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.1@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 00:59:43.704185 5148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:43.694609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:51587 (local address 127.3.75.62:33135)
0926 00:59:43.694682 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:43.694685 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:43.694858 (+ 173us) server_negotiation.cc:407] Connection header received
0926 00:59:43.695406 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:43.695407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:43.695420 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:43.695492 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:43.696433 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:43.697207 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:43.698855 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:43.699196 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:43.699253 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:43.701944 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:43.701945 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:43.701945 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 00:59:43.701951 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:43.703146 (+ 1195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:43.703486 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:43.703487 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:43.703487 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:43.703536 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:43.703731 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:43.703731 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:43.703731 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:43.703822 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:43.703870 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:43.704066 (+ 196us) server_negotiation.cc:299] Negotiation successful
0926 00:59:43.704125 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 00:59:43.704401 5195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:43.694572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:33135 (local address 127.3.75.1:51587)
0926 00:59:43.694749 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:43.694788 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:43.695299 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:43.695608 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:43.695614 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:43.696017 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0926 00:59:43.696326 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:43.696332 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:43.697309 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:43.697311 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:43.698765 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:43.698769 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:43.699286 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:43.699287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:43.699452 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:43.700028 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 00:59:43.700039 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 00:59:43.701855 (+ 1816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 00:59:43.703210 (+ 1355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 00:59:43.703215 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 00:59:43.703217 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 00:59:43.703416 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 00:59:43.703589 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 00:59:43.703590 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 00:59:43.703590 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 00:59:43.703680 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 00:59:43.703870 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 00:59:43.703874 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 00:59:43.703973 (+ 99us) client_negotiation.cc:769] Sending connection context
0926 00:59:43.704057 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 00:59:43.704162 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":100,"threads_started":1}
I20260926 00:59:43.704879 5302 heartbeater.cc:347] Connected to a master server at 127.3.75.62:33135
I20260926 00:59:43.704959 5302 heartbeater.cc:464] Registering TS with master...
I20260926 00:59:43.705153 5302 heartbeater.cc:511] Master 127.3.75.62:33135 requested a full tablet report, sending...
I20260926 00:59:43.705686 5098 ts_manager.cc:194] Registered new tserver with Master: 5ea28b33245c4862a7c4dfd2230860f7 (127.3.75.1:40211)
I20260926 00:59:43.706363 5098 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.75.1@KRBTEST.COM'} at 127.3.75.1:51587
I20260926 00:59:43.710456 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 5171
I20260926 00:59:43.710592 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/ts-0/wal/instance
I20260926 00:59:43.710755 3372 external_mini_cluster.cc:1428] Setting key 45f92523d578b69f0b07da383b474fc0
I20260926 00:59:43.711149 5148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:43.705849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:48925 (local address 127.3.75.62:33135)
0926 00:59:43.705904 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:43.705906 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:43.706047 (+ 141us) server_negotiation.cc:407] Connection header received
0926 00:59:43.706149 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:43.706150 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:43.706161 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:43.706216 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:43.706537 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:43.707130 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:43.707815 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:43.708010 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:43.708077 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:43.708926 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:43.708927 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:43.708928 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 00:59:43.708934 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:43.709982 (+ 1048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:43.710361 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:43.710362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:43.710363 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:43.710423 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:43.710628 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:43.710629 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:43.710630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:43.710736 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:43.710825 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:43.710988 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 00:59:43.711065 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 00:59:43.711165 5308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:43.705845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:33135 (local address 127.3.75.1:48925)
0926 00:59:43.705976 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:43.705985 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:43.706075 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:43.706221 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:43.706222 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:43.706393 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0926 00:59:43.706465 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:43.706467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:43.707219 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:43.707220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:43.707722 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:43.707723 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:43.708087 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:43.708088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:43.708160 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:43.708556 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 00:59:43.708561 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 00:59:43.708814 (+ 253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 00:59:43.710074 (+ 1260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 00:59:43.710075 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 00:59:43.710076 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 00:59:43.710284 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 00:59:43.710495 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 00:59:43.710496 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 00:59:43.710496 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 00:59:43.710574 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 00:59:43.710829 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 00:59:43.710830 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 00:59:43.710877 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 00:59:43.710962 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 00:59:43.711066 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":81,"threads_started":1}
WARNING: no policy specified for kudu/127.3.75.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab.
Entry for principal kudu/127.3.75.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab.
I20260926 00:59:43.734078 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.2:0
--local_ip_for_outbound_sockets=127.3.75.2
--webserver_interface=127.3.75.2
--webserver_port=0
--tserver_master_addrs=127.3.75.62:33135
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab
--principal=kudu/127.3.75.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:37213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 00:59:43.814626 5311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 00:59:43.814806 5311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 00:59:43.814831 5311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 00:59:43.815989 5311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 00:59:43.816037 5311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 00:59:43.816057 5311 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 00:59:43.816069 5311 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 00:59:43.816087 5311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.2
I20260926 00:59:43.817402 5311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:37213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab
--principal=kudu/127.3.75.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.75.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.75.62:33135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.2
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 00:59:43.817575 5311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 00:59:43.817730 5311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 00:59:43.822429 5316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:43.822441 5317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:43.822464 5319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 00:59:43.822666 5311 server_base.cc:1034] running on GCE node
I20260926 00:59:43.822834 5311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 00:59:43.823100 5311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 00:59:43.824265 5311 hybrid_clock.cc:648] HybridClock initialized: now 1790384383824252 us; error 28 us; skew 500 ppm
Sep 26 00:59:43 dist-test-slave-7v10 krb5kdc[5054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384383, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 00:59:43.826668 5311 init.cc:377] Logged in from keytab as kudu/127.3.75.2@KRBTEST.COM (short username kudu)
I20260926 00:59:43.827143 5311 webserver.cc:460] Webserver started at http://127.3.75.2:44249/ using document root <none> and password file <none>
I20260926 00:59:43.827319 5311 fs_manager.cc:359] Metadata directory not provided
I20260926 00:59:43.827365 5311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 00:59:43.827459 5311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 00:59:43.828461 5311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/ts-1/data/instance:
uuid: "3bec1377543d433e832f77e9ebc6d740"
format_stamp: "Formatted at 2026-09-26 00:59:43 on dist-test-slave-7v10"
server_key: "c7e735c1c7e8ff1500ff2222273b3ee9"
server_key_iv: "554daf02e0be0ca0a170ee8174acbe07"
server_key_version: "encryptionkey@0"
I20260926 00:59:43.828858 5311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/ts-1/wal/instance:
uuid: "3bec1377543d433e832f77e9ebc6d740"
format_stamp: "Formatted at 2026-09-26 00:59:43 on dist-test-slave-7v10"
server_key: "c7e735c1c7e8ff1500ff2222273b3ee9"
server_key_iv: "554daf02e0be0ca0a170ee8174acbe07"
server_key_version: "encryptionkey@0"
I20260926 00:59:43.830313 5311 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 00:59:43.831017 5326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:43.831195 5311 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 00:59:43.831282 5311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/ts-1/wal
uuid: "3bec1377543d433e832f77e9ebc6d740"
format_stamp: "Formatted at 2026-09-26 00:59:43 on dist-test-slave-7v10"
server_key: "c7e735c1c7e8ff1500ff2222273b3ee9"
server_key_iv: "554daf02e0be0ca0a170ee8174acbe07"
server_key_version: "encryptionkey@0"
I20260926 00:59:43.831382 5311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 00:59:43.840131 5311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 00:59:43.841789 5311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 00:59:43.841941 5311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 00:59:43.842447 5311 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 00:59:43.842485 5311 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:43.842505 5311 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 00:59:43.842518 5311 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 00:59:43 dist-test-slave-7v10 krb5kdc[5054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384383, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.2@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 00:59:43.849867 5311 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.2:41097
I20260926 00:59:43.850152 5441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.2:41097 every 8 connection(s)
I20260926 00:59:43.850191 5311 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
I20260926 00:59:43.852114 5148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:43.842829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:39955 (local address 127.3.75.62:33135)
0926 00:59:43.842921 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:43.842923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:43.843137 (+ 214us) server_negotiation.cc:407] Connection header received
0926 00:59:43.843561 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:43.843563 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:43.843577 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:43.843664 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:43.844203 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:43.844708 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:43.845765 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:43.845979 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:43.846065 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:43.849044 (+ 2979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:43.849045 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:43.849046 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 00:59:43.849053 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:43.850701 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:43.851165 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:43.851167 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:43.851167 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:43.851228 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:43.851477 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:43.851478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:43.851478 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:43.851592 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:43.851666 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:43.851933 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 00:59:43.852027 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 00:59:43.852252 5335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:43.842808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:33135 (local address 127.3.75.2:39955)
0926 00:59:43.843002 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:43.843040 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:43.843469 (+ 429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:43.843711 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:43.843714 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:43.843923 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0926 00:59:43.844107 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:43.844111 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:43.844815 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:43.844816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:43.845674 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:43.845676 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:43.846075 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:43.846076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:43.846188 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:43.846702 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 00:59:43.846712 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 00:59:43.848940 (+ 2228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 00:59:43.850785 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 00:59:43.850791 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 00:59:43.850793 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 00:59:43.851077 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 00:59:43.851295 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 00:59:43.851296 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 00:59:43.851296 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 00:59:43.851405 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 00:59:43.851657 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 00:59:43.851663 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 00:59:43.851797 (+ 134us) client_negotiation.cc:769] Sending connection context
0926 00:59:43.851896 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 00:59:43.852043 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":119,"threads_started":1}
I20260926 00:59:43.852705 5442 heartbeater.cc:347] Connected to a master server at 127.3.75.62:33135
I20260926 00:59:43.852804 5442 heartbeater.cc:464] Registering TS with master...
I20260926 00:59:43.852972 5442 heartbeater.cc:511] Master 127.3.75.62:33135 requested a full tablet report, sending...
I20260926 00:59:43.853329 5099 ts_manager.cc:194] Registered new tserver with Master: 3bec1377543d433e832f77e9ebc6d740 (127.3.75.2:41097)
I20260926 00:59:43.853789 5099 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.75.2@KRBTEST.COM'} at 127.3.75.2:39955
I20260926 00:59:43.858464 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 5311
I20260926 00:59:43.858595 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/ts-1/wal/instance
I20260926 00:59:43.858922 3372 external_mini_cluster.cc:1428] Setting key edcd1febedc2d53f2ad508080d1114c3
I20260926 00:59:43.859090 5148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:43.853754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:47227 (local address 127.3.75.62:33135)
0926 00:59:43.853794 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:43.853796 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:43.853921 (+ 125us) server_negotiation.cc:407] Connection header received
0926 00:59:43.854037 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:43.854038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:43.854046 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:43.854118 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:43.854438 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:43.854926 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:43.855590 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:43.855785 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:43.855855 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:43.856687 (+ 832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:43.856688 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:43.856689 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 00:59:43.856695 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:43.857811 (+ 1116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:43.858166 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:43.858167 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:43.858167 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:43.858216 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:43.858434 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:43.858435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:43.858435 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:43.858644 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:43.858711 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:43.858931 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 00:59:43.859005 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 00:59:43.859102 5448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:43.853668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:33135 (local address 127.3.75.2:47227)
0926 00:59:43.853800 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:43.853812 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:43.853982 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:43.854137 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:43.854138 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:43.854314 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0926 00:59:43.854358 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:43.854359 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:43.855008 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:43.855009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:43.855497 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:43.855499 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:43.855854 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:43.855855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:43.855916 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:43.856327 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 00:59:43.856332 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 00:59:43.856587 (+ 255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 00:59:43.857903 (+ 1316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 00:59:43.857904 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 00:59:43.857904 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 00:59:43.858071 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 00:59:43.858289 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 00:59:43.858290 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 00:59:43.858290 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 00:59:43.858367 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 00:59:43.858730 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 00:59:43.858731 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 00:59:43.858783 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 00:59:43.858896 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 00:59:43.858996 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":80,"threads_started":1}
WARNING: no policy specified for kudu/127.3.75.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab.
Entry for principal kudu/127.3.75.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab.
I20260926 00:59:43.881948 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.3:0
--local_ip_for_outbound_sockets=127.3.75.3
--webserver_interface=127.3.75.3
--webserver_port=0
--tserver_master_addrs=127.3.75.62:33135
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab
--principal=kudu/127.3.75.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:37213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 00:59:43.960777 5451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 00:59:43.960974 5451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 00:59:43.960997 5451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 00:59:43.962085 5451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 00:59:43.962118 5451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 00:59:43.962138 5451 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 00:59:43.962150 5451 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 00:59:43.962167 5451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.3
I20260926 00:59:43.963526 5451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:37213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab
--principal=kudu/127.3.75.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.75.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.75.62:33135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.3
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 00:59:43.963711 5451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 00:59:43.963871 5451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 00:59:43.968648 5459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:43.968704 5456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:43.968664 5457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 00:59:43.968887 5451 server_base.cc:1034] running on GCE node
I20260926 00:59:43.969107 5451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 00:59:43.969339 5451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 00:59:43.970498 5451 hybrid_clock.cc:648] HybridClock initialized: now 1790384383970489 us; error 28 us; skew 500 ppm
Sep 26 00:59:43 dist-test-slave-7v10 krb5kdc[5054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384383, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 00:59:43.972916 5451 init.cc:377] Logged in from keytab as kudu/127.3.75.3@KRBTEST.COM (short username kudu)
I20260926 00:59:43.973351 5451 webserver.cc:460] Webserver started at http://127.3.75.3:33739/ using document root <none> and password file <none>
I20260926 00:59:43.973528 5451 fs_manager.cc:359] Metadata directory not provided
I20260926 00:59:43.973572 5451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 00:59:43.973677 5451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 00:59:43.974666 5451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/ts-2/data/instance:
uuid: "668165b874ca4fcb8714fa15f91de91b"
format_stamp: "Formatted at 2026-09-26 00:59:43 on dist-test-slave-7v10"
server_key: "748cceadbfcb0ac74d3724d0127f36d4"
server_key_iv: "629d687eaf8b8977124134b535311138"
server_key_version: "encryptionkey@0"
I20260926 00:59:43.975092 5451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/ts-2/wal/instance:
uuid: "668165b874ca4fcb8714fa15f91de91b"
format_stamp: "Formatted at 2026-09-26 00:59:43 on dist-test-slave-7v10"
server_key: "748cceadbfcb0ac74d3724d0127f36d4"
server_key_iv: "629d687eaf8b8977124134b535311138"
server_key_version: "encryptionkey@0"
I20260926 00:59:43.976630 5451 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 00:59:43.977541 5466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:43.977707 5451 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 00:59:43.977793 5451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/ts-2/wal
uuid: "668165b874ca4fcb8714fa15f91de91b"
format_stamp: "Formatted at 2026-09-26 00:59:43 on dist-test-slave-7v10"
server_key: "748cceadbfcb0ac74d3724d0127f36d4"
server_key_iv: "629d687eaf8b8977124134b535311138"
server_key_version: "encryptionkey@0"
I20260926 00:59:43.977864 5451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 00:59:44.005066 5451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 00:59:44.007258 5451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 00:59:44.007428 5451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 00:59:44.007958 5451 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 00:59:44.007994 5451 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:44.008021 5451 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 00:59:44.008069 5451 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:44.015008 5451 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.3:41173
I20260926 00:59:44.015101 5581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.3:41173 every 8 connection(s)
I20260926 00:59:44.015378 5451 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
Sep 26 00:59:44 dist-test-slave-7v10 krb5kdc[5054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384383, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.3@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 00:59:44.016548 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 5451
I20260926 00:59:44.016654 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/ts-2/wal/instance
I20260926 00:59:44.016856 3372 external_mini_cluster.cc:1428] Setting key 5ea6e48795e120ed671d0efa38551cfe
I20260926 00:59:44.018756 5148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:44.008507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.3:47335 (local address 127.3.75.62:33135)
0926 00:59:44.008554 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:44.008556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:44.008819 (+ 263us) server_negotiation.cc:407] Connection header received
0926 00:59:44.009350 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:44.009351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:44.009364 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:44.009435 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:44.010407 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:44.011274 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:44.012986 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:44.013254 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:44.013307 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:44.015943 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:44.015944 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:44.015944 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 00:59:44.015949 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:44.017419 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:44.017793 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:44.017793 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:44.017794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:44.017853 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:44.018088 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:44.018089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:44.018089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:44.018198 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:44.018281 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:44.018569 (+ 288us) server_negotiation.cc:299] Negotiation successful
0926 00:59:44.018658 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 00:59:44.018959 5475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:44.008527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:33135 (local address 127.3.75.3:47335)
0926 00:59:44.008703 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:44.008737 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:44.009234 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:44.009563 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:44.009569 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:44.009954 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0926 00:59:44.010286 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:44.010294 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:44.011390 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:44.011391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:44.012882 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:44.012885 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:44.013337 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:44.013338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:44.013528 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:44.014066 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 00:59:44.014085 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 00:59:44.015861 (+ 1776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 00:59:44.017505 (+ 1644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 00:59:44.017511 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 00:59:44.017513 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 00:59:44.017710 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 00:59:44.017917 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 00:59:44.017918 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 00:59:44.017918 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 00:59:44.018020 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 00:59:44.018284 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 00:59:44.018290 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 00:59:44.018421 (+ 131us) client_negotiation.cc:769] Sending connection context
0926 00:59:44.018538 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 00:59:44.018697 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":99,"threads_started":1}
I20260926 00:59:44.019337 5582 heartbeater.cc:347] Connected to a master server at 127.3.75.62:33135
I20260926 00:59:44.019418 5582 heartbeater.cc:464] Registering TS with master...
I20260926 00:59:44.019601 5582 heartbeater.cc:511] Master 127.3.75.62:33135 requested a full tablet report, sending...
I20260926 00:59:44.019954 5098 ts_manager.cc:194] Registered new tserver with Master: 668165b874ca4fcb8714fa15f91de91b (127.3.75.3:41173)
I20260926 00:59:44.020440 5098 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.75.3@KRBTEST.COM'} at 127.3.75.3:47335
I20260926 00:59:44.025331 5148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:44.020566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.3:60361 (local address 127.3.75.62:33135)
0926 00:59:44.020600 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:44.020602 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:44.020608 (+ 6us) server_negotiation.cc:407] Connection header received
0926 00:59:44.020624 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:44.020624 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:44.020632 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:44.020682 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:44.021065 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:44.021524 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:44.022157 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:44.022327 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:44.022377 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:44.023173 (+ 796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:44.023174 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:44.023174 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 00:59:44.023180 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:44.024554 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:44.024821 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:44.024822 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:44.024822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:44.024866 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:44.025007 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:44.025007 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:44.025007 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:44.025087 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:44.025129 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:44.025228 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 00:59:44.025278 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 00:59:44.025348 5588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:44.020256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:33135 (local address 127.3.75.3:60361)
0926 00:59:44.020377 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:44.020387 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:44.020445 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:44.020718 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:44.020719 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:44.020932 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0926 00:59:44.020986 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:44.020988 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:44.021611 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:44.021611 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:44.022087 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:44.022088 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:44.022390 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:44.022390 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:44.022444 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:44.022781 (+ 337us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 00:59:44.022785 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 00:59:44.023107 (+ 322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 00:59:44.024620 (+ 1513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 00:59:44.024621 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 00:59:44.024621 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 00:59:44.024761 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 00:59:44.024920 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 00:59:44.024920 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 00:59:44.024920 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 00:59:44.024965 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 00:59:44.025133 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 00:59:44.025133 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 00:59:44.025164 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 00:59:44.025213 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 00:59:44.025282 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":74,"threads_started":1}
I20260926 00:59:44.028343 3372 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 00:59:44 dist-test-slave-7v10 krb5kdc[5054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 00:59:44 dist-test-slave-7v10 krb5kdc[5054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 00:59:44.049584 5148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:44.043385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58220 (local address 127.3.75.62:33135)
0926 00:59:44.043450 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:44.043452 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:44.043461 (+ 9us) server_negotiation.cc:407] Connection header received
0926 00:59:44.043488 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:44.043489 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:44.043497 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:44.043545 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:44.044048 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:44.044533 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:44.045162 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:44.045328 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:44.045376 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:44.047344 (+ 1968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:44.047345 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:44.047345 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 00:59:44.047351 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:44.048684 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:44.048969 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:44.048969 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:44.048970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:44.049014 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:44.049153 (+ 139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:44.049154 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:44.049154 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:44.049235 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:44.049300 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:44.049377 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 00:59:44.049445 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 00:59:44.051034 5098 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58220:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 00:59:44.052596 5098 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 00:59:44.059672 5601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:44.055071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54370 (local address 127.3.75.3:41173)
0926 00:59:44.055317 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:44.055322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:44.055554 (+ 232us) server_negotiation.cc:407] Connection header received
0926 00:59:44.055789 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:44.055792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:44.055807 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:44.055878 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 00:59:44.056097 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:44.057152 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:44.058419 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:44.059199 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:44.059267 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:44.059319 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:44.059537 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 00:59:44.059572 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":73,"threads_started":1}
I20260926 00:59:44.059731 5604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:44.055317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56984 (local address 127.3.75.1:40211)
0926 00:59:44.055545 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:44.055549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:44.055559 (+ 10us) server_negotiation.cc:407] Connection header received
0926 00:59:44.055603 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:44.055605 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:44.055617 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:44.055718 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 00:59:44.055890 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:44.057043 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:44.058673 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:44.059355 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:44.059453 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:44.059517 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:44.059595 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 00:59:44.059634 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":92,"threads_started":1}
I20260926 00:59:44.059718 5599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:44.054963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.1:40211 (local address 127.0.0.1:56984)
0926 00:59:44.055235 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:44.055274 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:44.055336 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:44.055735 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:44.055738 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:44.055741 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 00:59:44.055805 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:44.055807 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:44.057140 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:44.057141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:44.058577 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:44.058580 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:44.059456 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:44.059457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:44.059518 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:44.059523 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 00:59:44.059582 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 00:59:44.059640 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":61,"thread_start_us":56,"threads_started":1}
I20260926 00:59:44.060055 5602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:44.054983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.3:41173 (local address 127.0.0.1:54370)
0926 00:59:44.055490 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:44.055500 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:44.055622 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:44.055958 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:44.055958 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:44.055960 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 00:59:44.056015 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:44.056016 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:44.057211 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:44.057212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:44.058312 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:44.058315 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:44.059351 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:44.059352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:44.059427 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:44.059441 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 00:59:44.059869 (+ 428us) client_negotiation.cc:240] Negotiation successful
0926 00:59:44.059909 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"mutex_wait_us":49,"thread_start_us":74,"threads_started":1}
I20260926 00:59:44.060626 5236 tablet_service.cc:1467] Processing CreateTablet for tablet f0642df07a124ae08ef4698214f12eed (DEFAULT_TABLE table=test-table [id=b3ba97112ab04f3da84d80fe9e605e46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 00:59:44.060685 5516 tablet_service.cc:1467] Processing CreateTablet for tablet f0642df07a124ae08ef4698214f12eed (DEFAULT_TABLE table=test-table [id=b3ba97112ab04f3da84d80fe9e605e46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 00:59:44.060982 5236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0642df07a124ae08ef4698214f12eed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 00:59:44.061069 5516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0642df07a124ae08ef4698214f12eed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 00:59:44.061161 5600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:44.055164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.2:41097 (local address 127.0.0.1:40564)
0926 00:59:44.055805 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:44.055815 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:44.055864 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:44.057236 (+ 1372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:44.057236 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:44.057238 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 00:59:44.057271 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:44.057272 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:44.058514 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:44.058515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:44.059657 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:44.059659 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:44.060913 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:44.060914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:44.060972 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:44.060975 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 00:59:44.061037 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 00:59:44.061076 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":69,"threads_started":1}
I20260926 00:59:44.061556 5603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:44.055284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40564 (local address 127.3.75.2:41097)
0926 00:59:44.055508 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:44.055514 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:44.055924 (+ 410us) server_negotiation.cc:407] Connection header received
0926 00:59:44.055985 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:44.055988 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:44.056004 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:44.056083 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 00:59:44.057334 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:44.058424 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:44.059874 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:44.060751 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:44.060823 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:44.060876 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:44.061119 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 00:59:44.061451 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":116,"threads_started":1}
I20260926 00:59:44.062327 5376 tablet_service.cc:1467] Processing CreateTablet for tablet f0642df07a124ae08ef4698214f12eed (DEFAULT_TABLE table=test-table [id=b3ba97112ab04f3da84d80fe9e605e46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 00:59:44.062642 5376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0642df07a124ae08ef4698214f12eed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 00:59:44.063751 5605 tablet_bootstrap.cc:492] T f0642df07a124ae08ef4698214f12eed P 668165b874ca4fcb8714fa15f91de91b: Bootstrap starting.
I20260926 00:59:44.064554 5605 tablet_bootstrap.cc:654] T f0642df07a124ae08ef4698214f12eed P 668165b874ca4fcb8714fa15f91de91b: Neither blocks nor log segments found. Creating new log.
I20260926 00:59:44.064688 5606 tablet_bootstrap.cc:492] T f0642df07a124ae08ef4698214f12eed P 5ea28b33245c4862a7c4dfd2230860f7: Bootstrap starting.
I20260926 00:59:44.064885 5605 log.cc:824] T f0642df07a124ae08ef4698214f12eed P 668165b874ca4fcb8714fa15f91de91b: Log is configured to *not* fsync() on all Append() calls
I20260926 00:59:44.065291 5606 tablet_bootstrap.cc:654] T f0642df07a124ae08ef4698214f12eed P 5ea28b33245c4862a7c4dfd2230860f7: Neither blocks nor log segments found. Creating new log.
I20260926 00:59:44.065608 5606 log.cc:824] T f0642df07a124ae08ef4698214f12eed P 5ea28b33245c4862a7c4dfd2230860f7: Log is configured to *not* fsync() on all Append() calls
I20260926 00:59:44.065727 5605 tablet_bootstrap.cc:492] T f0642df07a124ae08ef4698214f12eed P 668165b874ca4fcb8714fa15f91de91b: No bootstrap required, opened a new log
I20260926 00:59:44.065785 5605 ts_tablet_manager.cc:1397] T f0642df07a124ae08ef4698214f12eed P 668165b874ca4fcb8714fa15f91de91b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 00:59:44.066326 5606 tablet_bootstrap.cc:492] T f0642df07a124ae08ef4698214f12eed P 5ea28b33245c4862a7c4dfd2230860f7: No bootstrap required, opened a new log
I20260926 00:59:44.066390 5606 ts_tablet_manager.cc:1397] T f0642df07a124ae08ef4698214f12eed P 5ea28b33245c4862a7c4dfd2230860f7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 00:59:44.067006 5609 tablet_bootstrap.cc:492] T f0642df07a124ae08ef4698214f12eed P 3bec1377543d433e832f77e9ebc6d740: Bootstrap starting.
I20260926 00:59:44.067610 5609 tablet_bootstrap.cc:654] T f0642df07a124ae08ef4698214f12eed P 3bec1377543d433e832f77e9ebc6d740: Neither blocks nor log segments found. Creating new log.
I20260926 00:59:44.067899 5609 log.cc:824] T f0642df07a124ae08ef4698214f12eed P 3bec1377543d433e832f77e9ebc6d740: Log is configured to *not* fsync() on all Append() calls
I20260926 00:59:44.068085 5605 raft_consensus.cc:348] T f0642df07a124ae08ef4698214f12eed P 668165b874ca4fcb8714fa15f91de91b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea28b33245c4862a7c4dfd2230860f7" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 40211 } } peers { permanent_uuid: "668165b874ca4fcb8714fa15f91de91b" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41173 } } peers { permanent_uuid: "3bec1377543d433e832f77e9ebc6d740" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41097 } }
I20260926 00:59:44.068257 5605 raft_consensus.cc:374] T f0642df07a124ae08ef4698214f12eed P 668165b874ca4fcb8714fa15f91de91b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 00:59:44.068295 5605 raft_consensus.cc:729] T f0642df07a124ae08ef4698214f12eed P 668165b874ca4fcb8714fa15f91de91b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668165b874ca4fcb8714fa15f91de91b, State: Initialized, Role: FOLLOWER
I20260926 00:59:44.068344 5606 raft_consensus.cc:348] T f0642df07a124ae08ef4698214f12eed P 5ea28b33245c4862a7c4dfd2230860f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea28b33245c4862a7c4dfd2230860f7" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 40211 } } peers { permanent_uuid: "668165b874ca4fcb8714fa15f91de91b" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41173 } } peers { permanent_uuid: "3bec1377543d433e832f77e9ebc6d740" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41097 } }
I20260926 00:59:44.068445 5605 consensus_queue.cc:260] T f0642df07a124ae08ef4698214f12eed P 668165b874ca4fcb8714fa15f91de91b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea28b33245c4862a7c4dfd2230860f7" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 40211 } } peers { permanent_uuid: "668165b874ca4fcb8714fa15f91de91b" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41173 } } peers { permanent_uuid: "3bec1377543d433e832f77e9ebc6d740" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41097 } }
I20260926 00:59:44.068491 5606 raft_consensus.cc:374] T f0642df07a124ae08ef4698214f12eed P 5ea28b33245c4862a7c4dfd2230860f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 00:59:44.068527 5606 raft_consensus.cc:729] T f0642df07a124ae08ef4698214f12eed P 5ea28b33245c4862a7c4dfd2230860f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ea28b33245c4862a7c4dfd2230860f7, State: Initialized, Role: FOLLOWER
I20260926 00:59:44.068621 5609 tablet_bootstrap.cc:492] T f0642df07a124ae08ef4698214f12eed P 3bec1377543d433e832f77e9ebc6d740: No bootstrap required, opened a new log
I20260926 00:59:44.068641 5606 consensus_queue.cc:260] T f0642df07a124ae08ef4698214f12eed P 5ea28b33245c4862a7c4dfd2230860f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea28b33245c4862a7c4dfd2230860f7" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 40211 } } peers { permanent_uuid: "668165b874ca4fcb8714fa15f91de91b" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41173 } } peers { permanent_uuid: "3bec1377543d433e832f77e9ebc6d740" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41097 } }
I20260926 00:59:44.068706 5609 ts_tablet_manager.cc:1397] T f0642df07a124ae08ef4698214f12eed P 3bec1377543d433e832f77e9ebc6d740: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 00:59:44.068711 5605 ts_tablet_manager.cc:1428] T f0642df07a124ae08ef4698214f12eed P 668165b874ca4fcb8714fa15f91de91b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 00:59:44.068841 5582 heartbeater.cc:503] Master 127.3.75.62:33135 was elected leader, sending a full tablet report...
I20260926 00:59:44.068907 5606 ts_tablet_manager.cc:1428] T f0642df07a124ae08ef4698214f12eed P 5ea28b33245c4862a7c4dfd2230860f7: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 00:59:44.068945 5302 heartbeater.cc:503] Master 127.3.75.62:33135 was elected leader, sending a full tablet report...
I20260926 00:59:44.070540 5609 raft_consensus.cc:348] T f0642df07a124ae08ef4698214f12eed P 3bec1377543d433e832f77e9ebc6d740 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea28b33245c4862a7c4dfd2230860f7" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 40211 } } peers { permanent_uuid: "668165b874ca4fcb8714fa15f91de91b" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41173 } } peers { permanent_uuid: "3bec1377543d433e832f77e9ebc6d740" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41097 } }
I20260926 00:59:44.070701 5609 raft_consensus.cc:374] T f0642df07a124ae08ef4698214f12eed P 3bec1377543d433e832f77e9ebc6d740 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 00:59:44.070744 5609 raft_consensus.cc:729] T f0642df07a124ae08ef4698214f12eed P 3bec1377543d433e832f77e9ebc6d740 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bec1377543d433e832f77e9ebc6d740, State: Initialized, Role: FOLLOWER
I20260926 00:59:44.070858 5609 consensus_queue.cc:260] T f0642df07a124ae08ef4698214f12eed P 3bec1377543d433e832f77e9ebc6d740 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea28b33245c4862a7c4dfd2230860f7" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 40211 } } peers { permanent_uuid: "668165b874ca4fcb8714fa15f91de91b" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41173 } } peers { permanent_uuid: "3bec1377543d433e832f77e9ebc6d740" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41097 } }
I20260926 00:59:44.071147 5609 ts_tablet_manager.cc:1428] T f0642df07a124ae08ef4698214f12eed P 3bec1377543d433e832f77e9ebc6d740: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260926 00:59:44.071199 5442 heartbeater.cc:503] Master 127.3.75.62:33135 was elected leader, sending a full tablet report...
W20260926 00:59:44.100860 5443 tablet.cc:2369] T f0642df07a124ae08ef4698214f12eed P 3bec1377543d433e832f77e9ebc6d740: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 00:59:44.202085 5303 tablet.cc:2369] T f0642df07a124ae08ef4698214f12eed P 5ea28b33245c4862a7c4dfd2230860f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 00:59:44.265592 5613 raft_consensus.cc:482] T f0642df07a124ae08ef4698214f12eed P 3bec1377543d433e832f77e9ebc6d740 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 00:59:44.265723 5613 raft_consensus.cc:504] T f0642df07a124ae08ef4698214f12eed P 3bec1377543d433e832f77e9ebc6d740 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea28b33245c4862a7c4dfd2230860f7" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 40211 } } peers { permanent_uuid: "668165b874ca4fcb8714fa15f91de91b" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41173 } } peers { permanent_uuid: "3bec1377543d433e832f77e9ebc6d740" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41097 } }
W20260926 00:59:44.265929 5583 tablet.cc:2369] T f0642df07a124ae08ef4698214f12eed P 668165b874ca4fcb8714fa15f91de91b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 00:59:44.266165 5613 leader_election.cc:288] T f0642df07a124ae08ef4698214f12eed P 3bec1377543d433e832f77e9ebc6d740 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ea28b33245c4862a7c4dfd2230860f7 (127.3.75.1:40211), 668165b874ca4fcb8714fa15f91de91b (127.3.75.3:41173)
I20260926 00:59:44.268965 5335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:44.266358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.3:41173 (local address 127.3.75.2:53627)
0926 00:59:44.266417 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:44.266430 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:44.266484 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:44.266635 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:44.266636 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:44.266638 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 00:59:44.266702 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:44.266704 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:44.267400 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:44.267401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:44.268161 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:44.268163 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:44.268694 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:44.268695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:44.268782 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:44.268785 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 00:59:44.268859 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 00:59:44.268899 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":18}
I20260926 00:59:44.268963 5601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:44.266444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:53627 (local address 127.3.75.3:41173)
0926 00:59:44.266502 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:44.266504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:44.266514 (+ 10us) server_negotiation.cc:407] Connection header received
0926 00:59:44.266545 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:44.266546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:44.266557 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:44.266620 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 00:59:44.266786 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:44.267312 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:44.268255 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:44.268613 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:44.268696 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:44.268705 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:44.268866 (+ 161us) server_negotiation.cc:299] Negotiation successful
0926 00:59:44.268898 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 00:59:44.269059 5604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:44.266442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:55637 (local address 127.3.75.1:40211)
0926 00:59:44.266491 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:44.266493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:44.266698 (+ 205us) server_negotiation.cc:407] Connection header received
0926 00:59:44.266757 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:44.266758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:44.266770 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:44.266819 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 00:59:44.266992 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:44.267427 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:44.268336 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:44.268689 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:44.268779 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:44.268789 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:44.268946 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 00:59:44.268995 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 00:59:44.269083 5614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:44.266358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.1:40211 (local address 127.3.75.2:55637)
0926 00:59:44.266649 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:44.266658 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:44.266705 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:44.266822 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:44.266823 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:44.266824 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 00:59:44.266919 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:44.266921 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:44.267513 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:44.267514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:44.268243 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:44.268244 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:44.268781 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:44.268782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:44.268856 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:44.268866 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 00:59:44.268928 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 00:59:44.268965 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":110,"thread_start_us":110,"threads_started":1}
I20260926 00:59:44.269407 5536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0642df07a124ae08ef4698214f12eed" candidate_uuid: "3bec1377543d433e832f77e9ebc6d740" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668165b874ca4fcb8714fa15f91de91b" is_pre_election: true
I20260926 00:59:44.269565 5536 raft_consensus.cc:2454] T f0642df07a124ae08ef4698214f12eed P 668165b874ca4fcb8714fa15f91de91b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bec1377543d433e832f77e9ebc6d740 in term 0.
I20260926 00:59:44.269534 5256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0642df07a124ae08ef4698214f12eed" candidate_uuid: "3bec1377543d433e832f77e9ebc6d740" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ea28b33245c4862a7c4dfd2230860f7" is_pre_election: true
I20260926 00:59:44.269670 5256 raft_consensus.cc:2454] T f0642df07a124ae08ef4698214f12eed P 5ea28b33245c4862a7c4dfd2230860f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bec1377543d433e832f77e9ebc6d740 in term 0.
I20260926 00:59:44.269788 5328 leader_election.cc:302] T f0642df07a124ae08ef4698214f12eed P 3bec1377543d433e832f77e9ebc6d740 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bec1377543d433e832f77e9ebc6d740, 668165b874ca4fcb8714fa15f91de91b; no voters:
I20260926 00:59:44.269920 5613 raft_consensus.cc:2781] T f0642df07a124ae08ef4698214f12eed P 3bec1377543d433e832f77e9ebc6d740 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 00:59:44.269984 5613 raft_consensus.cc:482] T f0642df07a124ae08ef4698214f12eed P 3bec1377543d433e832f77e9ebc6d740 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 00:59:44.270007 5613 raft_consensus.cc:3037] T f0642df07a124ae08ef4698214f12eed P 3bec1377543d433e832f77e9ebc6d740 [term 0 FOLLOWER]: Advancing to term 1
I20260926 00:59:44.270964 5613 raft_consensus.cc:504] T f0642df07a124ae08ef4698214f12eed P 3bec1377543d433e832f77e9ebc6d740 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea28b33245c4862a7c4dfd2230860f7" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 40211 } } peers { permanent_uuid: "668165b874ca4fcb8714fa15f91de91b" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41173 } } peers { permanent_uuid: "3bec1377543d433e832f77e9ebc6d740" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41097 } }
I20260926 00:59:44.271121 5613 leader_election.cc:288] T f0642df07a124ae08ef4698214f12eed P 3bec1377543d433e832f77e9ebc6d740 [CANDIDATE]: Term 1 election: Requested vote from peers 5ea28b33245c4862a7c4dfd2230860f7 (127.3.75.1:40211), 668165b874ca4fcb8714fa15f91de91b (127.3.75.3:41173)
I20260926 00:59:44.271270 5536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0642df07a124ae08ef4698214f12eed" candidate_uuid: "3bec1377543d433e832f77e9ebc6d740" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668165b874ca4fcb8714fa15f91de91b"
I20260926 00:59:44.271268 5256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0642df07a124ae08ef4698214f12eed" candidate_uuid: "3bec1377543d433e832f77e9ebc6d740" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ea28b33245c4862a7c4dfd2230860f7"
I20260926 00:59:44.271343 5536 raft_consensus.cc:3037] T f0642df07a124ae08ef4698214f12eed P 668165b874ca4fcb8714fa15f91de91b [term 0 FOLLOWER]: Advancing to term 1
I20260926 00:59:44.271343 5256 raft_consensus.cc:3037] T f0642df07a124ae08ef4698214f12eed P 5ea28b33245c4862a7c4dfd2230860f7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 00:59:44.272169 5536 raft_consensus.cc:2454] T f0642df07a124ae08ef4698214f12eed P 668165b874ca4fcb8714fa15f91de91b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bec1377543d433e832f77e9ebc6d740 in term 1.
I20260926 00:59:44.272169 5256 raft_consensus.cc:2454] T f0642df07a124ae08ef4698214f12eed P 5ea28b33245c4862a7c4dfd2230860f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bec1377543d433e832f77e9ebc6d740 in term 1.
I20260926 00:59:44.272349 5329 leader_election.cc:302] T f0642df07a124ae08ef4698214f12eed P 3bec1377543d433e832f77e9ebc6d740 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bec1377543d433e832f77e9ebc6d740, 5ea28b33245c4862a7c4dfd2230860f7; no voters:
I20260926 00:59:44.272442 5613 raft_consensus.cc:2781] T f0642df07a124ae08ef4698214f12eed P 3bec1377543d433e832f77e9ebc6d740 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 00:59:44.272604 5613 raft_consensus.cc:686] T f0642df07a124ae08ef4698214f12eed P 3bec1377543d433e832f77e9ebc6d740 [term 1 LEADER]: Becoming Leader. State: Replica: 3bec1377543d433e832f77e9ebc6d740, State: Running, Role: LEADER
I20260926 00:59:44.272696 5613 consensus_queue.cc:237] T f0642df07a124ae08ef4698214f12eed P 3bec1377543d433e832f77e9ebc6d740 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea28b33245c4862a7c4dfd2230860f7" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 40211 } } peers { permanent_uuid: "668165b874ca4fcb8714fa15f91de91b" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41173 } } peers { permanent_uuid: "3bec1377543d433e832f77e9ebc6d740" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41097 } }
I20260926 00:59:44.273486 5098 catalog_manager.cc:5495] T f0642df07a124ae08ef4698214f12eed P 3bec1377543d433e832f77e9ebc6d740 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3bec1377543d433e832f77e9ebc6d740 (127.3.75.2). New cstate: current_term: 1 leader_uuid: "3bec1377543d433e832f77e9ebc6d740" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea28b33245c4862a7c4dfd2230860f7" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 40211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668165b874ca4fcb8714fa15f91de91b" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3bec1377543d433e832f77e9ebc6d740" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41097 } health_report { overall_health: HEALTHY } } }
Sep 26 00:59:44 dist-test-slave-7v10 krb5kdc[5054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384383, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.62@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 00:59:44.302090 5622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:44.295526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:33135 (local address 127.0.0.1:58226)
0926 00:59:44.295665 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:44.295674 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:44.295735 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:44.295855 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:44.295856 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:44.296186 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0926 00:59:44.296237 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:44.296239 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:44.296904 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:44.296905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:44.297512 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:44.297515 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:44.297911 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:44.297912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:44.297959 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:44.298454 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 00:59:44.298459 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 00:59:44.299852 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 00:59:44.301186 (+ 1334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 00:59:44.301192 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 00:59:44.301194 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 00:59:44.301407 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 00:59:44.301574 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 00:59:44.301574 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 00:59:44.301575 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 00:59:44.301631 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 00:59:44.301821 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 00:59:44.301826 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 00:59:44.301864 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 00:59:44.301915 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 00:59:44.302002 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":62,"threads_started":1}
I20260926 00:59:44.302138 5148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:44.295580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58226 (local address 127.3.75.62:33135)
0926 00:59:44.295665 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:44.295668 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:44.295738 (+ 70us) server_negotiation.cc:407] Connection header received
0926 00:59:44.295775 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:44.295776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:44.295784 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:44.295835 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:44.296307 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:44.296823 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:44.297653 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:44.297850 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:44.297899 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:44.299931 (+ 2032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:44.299932 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:44.299933 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 00:59:44.299938 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:44.301116 (+ 1178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:44.301469 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:44.301470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:44.301470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:44.301517 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:44.301681 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:44.301681 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:44.301682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:44.301770 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:44.301817 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:44.301946 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 00:59:44.302049 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":59}
I20260926 00:59:44.304162 5098 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:58226:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 00:59:44.304340 5098 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 00:59:44.305799 5236 tablet_service.cc:1467] Processing CreateTablet for tablet 8e0d65cda77944dfbf9acc18f4e9d041 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ef09043691374e839903a1187cdb56ff]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 00:59:44.305811 5516 tablet_service.cc:1467] Processing CreateTablet for tablet 8e0d65cda77944dfbf9acc18f4e9d041 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ef09043691374e839903a1187cdb56ff]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 00:59:44.305819 5376 tablet_service.cc:1467] Processing CreateTablet for tablet 8e0d65cda77944dfbf9acc18f4e9d041 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ef09043691374e839903a1187cdb56ff]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 00:59:44.305969 5236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e0d65cda77944dfbf9acc18f4e9d041. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 00:59:44.305968 5376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e0d65cda77944dfbf9acc18f4e9d041. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 00:59:44.305979 5516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e0d65cda77944dfbf9acc18f4e9d041. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 00:59:44.307257 5606 tablet_bootstrap.cc:492] T 8e0d65cda77944dfbf9acc18f4e9d041 P 5ea28b33245c4862a7c4dfd2230860f7: Bootstrap starting.
I20260926 00:59:44.307441 5605 tablet_bootstrap.cc:492] T 8e0d65cda77944dfbf9acc18f4e9d041 P 668165b874ca4fcb8714fa15f91de91b: Bootstrap starting.
I20260926 00:59:44.307678 5606 tablet_bootstrap.cc:654] T 8e0d65cda77944dfbf9acc18f4e9d041 P 5ea28b33245c4862a7c4dfd2230860f7: Neither blocks nor log segments found. Creating new log.
I20260926 00:59:44.308080 5605 tablet_bootstrap.cc:654] T 8e0d65cda77944dfbf9acc18f4e9d041 P 668165b874ca4fcb8714fa15f91de91b: Neither blocks nor log segments found. Creating new log.
I20260926 00:59:44.308386 5606 tablet_bootstrap.cc:492] T 8e0d65cda77944dfbf9acc18f4e9d041 P 5ea28b33245c4862a7c4dfd2230860f7: No bootstrap required, opened a new log
I20260926 00:59:44.308440 5606 ts_tablet_manager.cc:1397] T 8e0d65cda77944dfbf9acc18f4e9d041 P 5ea28b33245c4862a7c4dfd2230860f7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 00:59:44.308457 5609 tablet_bootstrap.cc:492] T 8e0d65cda77944dfbf9acc18f4e9d041 P 3bec1377543d433e832f77e9ebc6d740: Bootstrap starting.
I20260926 00:59:44.308593 5606 raft_consensus.cc:348] T 8e0d65cda77944dfbf9acc18f4e9d041 P 5ea28b33245c4862a7c4dfd2230860f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea28b33245c4862a7c4dfd2230860f7" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 40211 } } peers { permanent_uuid: "3bec1377543d433e832f77e9ebc6d740" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41097 } } peers { permanent_uuid: "668165b874ca4fcb8714fa15f91de91b" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41173 } }
I20260926 00:59:44.308679 5606 raft_consensus.cc:374] T 8e0d65cda77944dfbf9acc18f4e9d041 P 5ea28b33245c4862a7c4dfd2230860f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 00:59:44.308703 5606 raft_consensus.cc:729] T 8e0d65cda77944dfbf9acc18f4e9d041 P 5ea28b33245c4862a7c4dfd2230860f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ea28b33245c4862a7c4dfd2230860f7, State: Initialized, Role: FOLLOWER
I20260926 00:59:44.308746 5606 consensus_queue.cc:260] T 8e0d65cda77944dfbf9acc18f4e9d041 P 5ea28b33245c4862a7c4dfd2230860f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea28b33245c4862a7c4dfd2230860f7" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 40211 } } peers { permanent_uuid: "3bec1377543d433e832f77e9ebc6d740" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41097 } } peers { permanent_uuid: "668165b874ca4fcb8714fa15f91de91b" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41173 } }
I20260926 00:59:44.308876 5606 ts_tablet_manager.cc:1428] T 8e0d65cda77944dfbf9acc18f4e9d041 P 5ea28b33245c4862a7c4dfd2230860f7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:44.308878 5609 tablet_bootstrap.cc:654] T 8e0d65cda77944dfbf9acc18f4e9d041 P 3bec1377543d433e832f77e9ebc6d740: Neither blocks nor log segments found. Creating new log.
I20260926 00:59:44.309409 5612 tablet_replica.cc:439] T 8e0d65cda77944dfbf9acc18f4e9d041 P 5ea28b33245c4862a7c4dfd2230860f7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea28b33245c4862a7c4dfd2230860f7" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 40211 } } peers { permanent_uuid: "3bec1377543d433e832f77e9ebc6d740" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41097 } } peers { permanent_uuid: "668165b874ca4fcb8714fa15f91de91b" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41173 } } }
I20260926 00:59:44.309517 5612 tablet_replica.cc:442] T 8e0d65cda77944dfbf9acc18f4e9d041 P 5ea28b33245c4862a7c4dfd2230860f7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 00:59:44.309544 5609 tablet_bootstrap.cc:492] T 8e0d65cda77944dfbf9acc18f4e9d041 P 3bec1377543d433e832f77e9ebc6d740: No bootstrap required, opened a new log
I20260926 00:59:44.309584 5609 ts_tablet_manager.cc:1397] T 8e0d65cda77944dfbf9acc18f4e9d041 P 3bec1377543d433e832f77e9ebc6d740: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 00:59:44.309725 5609 raft_consensus.cc:348] T 8e0d65cda77944dfbf9acc18f4e9d041 P 3bec1377543d433e832f77e9ebc6d740 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea28b33245c4862a7c4dfd2230860f7" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 40211 } } peers { permanent_uuid: "3bec1377543d433e832f77e9ebc6d740" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41097 } } peers { permanent_uuid: "668165b874ca4fcb8714fa15f91de91b" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41173 } }
I20260926 00:59:44.309825 5609 raft_consensus.cc:374] T 8e0d65cda77944dfbf9acc18f4e9d041 P 3bec1377543d433e832f77e9ebc6d740 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 00:59:44.309855 5609 raft_consensus.cc:729] T 8e0d65cda77944dfbf9acc18f4e9d041 P 3bec1377543d433e832f77e9ebc6d740 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bec1377543d433e832f77e9ebc6d740, State: Initialized, Role: FOLLOWER
I20260926 00:59:44.309897 5605 tablet_bootstrap.cc:492] T 8e0d65cda77944dfbf9acc18f4e9d041 P 668165b874ca4fcb8714fa15f91de91b: No bootstrap required, opened a new log
I20260926 00:59:44.309896 5609 consensus_queue.cc:260] T 8e0d65cda77944dfbf9acc18f4e9d041 P 3bec1377543d433e832f77e9ebc6d740 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea28b33245c4862a7c4dfd2230860f7" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 40211 } } peers { permanent_uuid: "3bec1377543d433e832f77e9ebc6d740" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41097 } } peers { permanent_uuid: "668165b874ca4fcb8714fa15f91de91b" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41173 } }
I20260926 00:59:44.309947 5605 ts_tablet_manager.cc:1397] T 8e0d65cda77944dfbf9acc18f4e9d041 P 668165b874ca4fcb8714fa15f91de91b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 00:59:44.309988 5609 ts_tablet_manager.cc:1428] T 8e0d65cda77944dfbf9acc18f4e9d041 P 3bec1377543d433e832f77e9ebc6d740: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 00:59:44.310097 5605 raft_consensus.cc:348] T 8e0d65cda77944dfbf9acc18f4e9d041 P 668165b874ca4fcb8714fa15f91de91b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea28b33245c4862a7c4dfd2230860f7" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 40211 } } peers { permanent_uuid: "3bec1377543d433e832f77e9ebc6d740" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41097 } } peers { permanent_uuid: "668165b874ca4fcb8714fa15f91de91b" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41173 } }
I20260926 00:59:44.310128 5613 tablet_replica.cc:439] T 8e0d65cda77944dfbf9acc18f4e9d041 P 3bec1377543d433e832f77e9ebc6d740: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea28b33245c4862a7c4dfd2230860f7" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 40211 } } peers { permanent_uuid: "3bec1377543d433e832f77e9ebc6d740" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41097 } } peers { permanent_uuid: "668165b874ca4fcb8714fa15f91de91b" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41173 } } }
I20260926 00:59:44.310170 5605 raft_consensus.cc:374] T 8e0d65cda77944dfbf9acc18f4e9d041 P 668165b874ca4fcb8714fa15f91de91b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 00:59:44.310192 5605 raft_consensus.cc:729] T 8e0d65cda77944dfbf9acc18f4e9d041 P 668165b874ca4fcb8714fa15f91de91b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668165b874ca4fcb8714fa15f91de91b, State: Initialized, Role: FOLLOWER
I20260926 00:59:44.310195 5613 tablet_replica.cc:442] T 8e0d65cda77944dfbf9acc18f4e9d041 P 3bec1377543d433e832f77e9ebc6d740: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 00:59:44.310240 5605 consensus_queue.cc:260] T 8e0d65cda77944dfbf9acc18f4e9d041 P 668165b874ca4fcb8714fa15f91de91b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea28b33245c4862a7c4dfd2230860f7" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 40211 } } peers { permanent_uuid: "3bec1377543d433e832f77e9ebc6d740" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41097 } } peers { permanent_uuid: "668165b874ca4fcb8714fa15f91de91b" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41173 } }
I20260926 00:59:44.310338 5605 ts_tablet_manager.cc:1428] T 8e0d65cda77944dfbf9acc18f4e9d041 P 668165b874ca4fcb8714fa15f91de91b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 00:59:44.310472 5611 tablet_replica.cc:439] T 8e0d65cda77944dfbf9acc18f4e9d041 P 668165b874ca4fcb8714fa15f91de91b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea28b33245c4862a7c4dfd2230860f7" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 40211 } } peers { permanent_uuid: "3bec1377543d433e832f77e9ebc6d740" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41097 } } peers { permanent_uuid: "668165b874ca4fcb8714fa15f91de91b" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41173 } } }
I20260926 00:59:44.310573 5611 tablet_replica.cc:442] T 8e0d65cda77944dfbf9acc18f4e9d041 P 668165b874ca4fcb8714fa15f91de91b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 00:59:44.322301 5613 raft_consensus.cc:482] T 8e0d65cda77944dfbf9acc18f4e9d041 P 3bec1377543d433e832f77e9ebc6d740 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 00:59:44.322355 5613 raft_consensus.cc:504] T 8e0d65cda77944dfbf9acc18f4e9d041 P 3bec1377543d433e832f77e9ebc6d740 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea28b33245c4862a7c4dfd2230860f7" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 40211 } } peers { permanent_uuid: "3bec1377543d433e832f77e9ebc6d740" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41097 } } peers { permanent_uuid: "668165b874ca4fcb8714fa15f91de91b" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41173 } }
I20260926 00:59:44.322528 5613 leader_election.cc:288] T 8e0d65cda77944dfbf9acc18f4e9d041 P 3bec1377543d433e832f77e9ebc6d740 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ea28b33245c4862a7c4dfd2230860f7 (127.3.75.1:40211), 668165b874ca4fcb8714fa15f91de91b (127.3.75.3:41173)
I20260926 00:59:44.322685 5256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e0d65cda77944dfbf9acc18f4e9d041" candidate_uuid: "3bec1377543d433e832f77e9ebc6d740" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ea28b33245c4862a7c4dfd2230860f7" is_pre_election: true
I20260926 00:59:44.322764 5256 raft_consensus.cc:2454] T 8e0d65cda77944dfbf9acc18f4e9d041 P 5ea28b33245c4862a7c4dfd2230860f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bec1377543d433e832f77e9ebc6d740 in term 0.
I20260926 00:59:44.322726 5536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e0d65cda77944dfbf9acc18f4e9d041" candidate_uuid: "3bec1377543d433e832f77e9ebc6d740" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668165b874ca4fcb8714fa15f91de91b" is_pre_election: true
I20260926 00:59:44.322824 5536 raft_consensus.cc:2454] T 8e0d65cda77944dfbf9acc18f4e9d041 P 668165b874ca4fcb8714fa15f91de91b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bec1377543d433e832f77e9ebc6d740 in term 0.
I20260926 00:59:44.322921 5329 leader_election.cc:302] T 8e0d65cda77944dfbf9acc18f4e9d041 P 3bec1377543d433e832f77e9ebc6d740 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bec1377543d433e832f77e9ebc6d740, 5ea28b33245c4862a7c4dfd2230860f7; no voters:
I20260926 00:59:44.323024 5613 raft_consensus.cc:2781] T 8e0d65cda77944dfbf9acc18f4e9d041 P 3bec1377543d433e832f77e9ebc6d740 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 00:59:44.323072 5613 raft_consensus.cc:482] T 8e0d65cda77944dfbf9acc18f4e9d041 P 3bec1377543d433e832f77e9ebc6d740 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 00:59:44.323136 5613 raft_consensus.cc:3037] T 8e0d65cda77944dfbf9acc18f4e9d041 P 3bec1377543d433e832f77e9ebc6d740 [term 0 FOLLOWER]: Advancing to term 1
I20260926 00:59:44.323689 5613 raft_consensus.cc:504] T 8e0d65cda77944dfbf9acc18f4e9d041 P 3bec1377543d433e832f77e9ebc6d740 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea28b33245c4862a7c4dfd2230860f7" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 40211 } } peers { permanent_uuid: "3bec1377543d433e832f77e9ebc6d740" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41097 } } peers { permanent_uuid: "668165b874ca4fcb8714fa15f91de91b" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41173 } }
I20260926 00:59:44.323828 5613 leader_election.cc:288] T 8e0d65cda77944dfbf9acc18f4e9d041 P 3bec1377543d433e832f77e9ebc6d740 [CANDIDATE]: Term 1 election: Requested vote from peers 5ea28b33245c4862a7c4dfd2230860f7 (127.3.75.1:40211), 668165b874ca4fcb8714fa15f91de91b (127.3.75.3:41173)
I20260926 00:59:44.323933 5256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e0d65cda77944dfbf9acc18f4e9d041" candidate_uuid: "3bec1377543d433e832f77e9ebc6d740" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ea28b33245c4862a7c4dfd2230860f7"
I20260926 00:59:44.323971 5536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e0d65cda77944dfbf9acc18f4e9d041" candidate_uuid: "3bec1377543d433e832f77e9ebc6d740" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668165b874ca4fcb8714fa15f91de91b"
I20260926 00:59:44.324020 5536 raft_consensus.cc:3037] T 8e0d65cda77944dfbf9acc18f4e9d041 P 668165b874ca4fcb8714fa15f91de91b [term 0 FOLLOWER]: Advancing to term 1
I20260926 00:59:44.324021 5256 raft_consensus.cc:3037] T 8e0d65cda77944dfbf9acc18f4e9d041 P 5ea28b33245c4862a7c4dfd2230860f7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 00:59:44.324707 5256 raft_consensus.cc:2454] T 8e0d65cda77944dfbf9acc18f4e9d041 P 5ea28b33245c4862a7c4dfd2230860f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bec1377543d433e832f77e9ebc6d740 in term 1.
I20260926 00:59:44.324702 5536 raft_consensus.cc:2454] T 8e0d65cda77944dfbf9acc18f4e9d041 P 668165b874ca4fcb8714fa15f91de91b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bec1377543d433e832f77e9ebc6d740 in term 1.
I20260926 00:59:44.324865 5328 leader_election.cc:302] T 8e0d65cda77944dfbf9acc18f4e9d041 P 3bec1377543d433e832f77e9ebc6d740 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bec1377543d433e832f77e9ebc6d740, 668165b874ca4fcb8714fa15f91de91b; no voters:
I20260926 00:59:44.324949 5613 raft_consensus.cc:2781] T 8e0d65cda77944dfbf9acc18f4e9d041 P 3bec1377543d433e832f77e9ebc6d740 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 00:59:44.325006 5613 raft_consensus.cc:686] T 8e0d65cda77944dfbf9acc18f4e9d041 P 3bec1377543d433e832f77e9ebc6d740 [term 1 LEADER]: Becoming Leader. State: Replica: 3bec1377543d433e832f77e9ebc6d740, State: Running, Role: LEADER
I20260926 00:59:44.325057 5613 consensus_queue.cc:237] T 8e0d65cda77944dfbf9acc18f4e9d041 P 3bec1377543d433e832f77e9ebc6d740 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea28b33245c4862a7c4dfd2230860f7" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 40211 } } peers { permanent_uuid: "3bec1377543d433e832f77e9ebc6d740" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41097 } } peers { permanent_uuid: "668165b874ca4fcb8714fa15f91de91b" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41173 } }
I20260926 00:59:44.325346 5615 tablet_replica.cc:439] T 8e0d65cda77944dfbf9acc18f4e9d041 P 3bec1377543d433e832f77e9ebc6d740: TxnStatusTablet state changed. Reason: New leader 3bec1377543d433e832f77e9ebc6d740. Latest consensus state: current_term: 1 leader_uuid: "3bec1377543d433e832f77e9ebc6d740" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea28b33245c4862a7c4dfd2230860f7" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 40211 } } peers { permanent_uuid: "3bec1377543d433e832f77e9ebc6d740" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41097 } } peers { permanent_uuid: "668165b874ca4fcb8714fa15f91de91b" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41173 } } }
I20260926 00:59:44.325405 5615 tablet_replica.cc:442] T 8e0d65cda77944dfbf9acc18f4e9d041 P 3bec1377543d433e832f77e9ebc6d740: This TxnStatusTablet replica's current role is: LEADER
I20260926 00:59:44.325675 5627 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 00:59:44.325611 5099 catalog_manager.cc:5495] T 8e0d65cda77944dfbf9acc18f4e9d041 P 3bec1377543d433e832f77e9ebc6d740 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3bec1377543d433e832f77e9ebc6d740 (127.3.75.2). New cstate: current_term: 1 leader_uuid: "3bec1377543d433e832f77e9ebc6d740" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea28b33245c4862a7c4dfd2230860f7" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 40211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3bec1377543d433e832f77e9ebc6d740" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668165b874ca4fcb8714fa15f91de91b" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41173 } health_report { overall_health: UNKNOWN } } }
I20260926 00:59:44.325773 5627 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 00:59:44.332245 5603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:44.329402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40580 (local address 127.3.75.2:41097)
0926 00:59:44.329453 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:44.329455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:44.329463 (+ 8us) server_negotiation.cc:407] Connection header received
0926 00:59:44.329536 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:44.329537 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:44.329553 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:44.329602 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 00:59:44.329739 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:44.330368 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:44.331484 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:44.331666 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:44.331720 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:44.331960 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 00:59:44.332045 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 00:59:44.332098 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:44.332183 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 00:59:44.332199 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 00:59:44.332307 5622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:44.329332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.2:41097 (local address 127.0.0.1:40580)
0926 00:59:44.329402 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:44.329413 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:44.329452 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:44.329628 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:44.329629 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:44.329630 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 00:59:44.329674 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:44.329677 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:44.330441 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:44.330442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:44.331407 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:44.331410 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:44.331746 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:44.331748 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:44.331840 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:44.331859 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 00:59:44.332112 (+ 253us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 00:59:44.332120 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 00:59:44.332176 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 00:59:44.332227 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 00:59:44.333909 5256 raft_consensus.cc:1261] T 8e0d65cda77944dfbf9acc18f4e9d041 P 5ea28b33245c4862a7c4dfd2230860f7 [term 1 FOLLOWER]: Refusing update from remote peer 3bec1377543d433e832f77e9ebc6d740: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 00:59:44.334189 5536 raft_consensus.cc:1261] T 8e0d65cda77944dfbf9acc18f4e9d041 P 668165b874ca4fcb8714fa15f91de91b [term 1 FOLLOWER]: Refusing update from remote peer 3bec1377543d433e832f77e9ebc6d740: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 00:59:44.334199 5613 consensus_queue.cc:1035] T 8e0d65cda77944dfbf9acc18f4e9d041 P 3bec1377543d433e832f77e9ebc6d740 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ea28b33245c4862a7c4dfd2230860f7" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 40211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 00:59:44.334479 5613 consensus_queue.cc:1035] T 8e0d65cda77944dfbf9acc18f4e9d041 P 3bec1377543d433e832f77e9ebc6d740 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668165b874ca4fcb8714fa15f91de91b" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 00:59:44.335367 5612 tablet_replica.cc:439] T 8e0d65cda77944dfbf9acc18f4e9d041 P 5ea28b33245c4862a7c4dfd2230860f7: TxnStatusTablet state changed. Reason: New leader 3bec1377543d433e832f77e9ebc6d740. Latest consensus state: current_term: 1 leader_uuid: "3bec1377543d433e832f77e9ebc6d740" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea28b33245c4862a7c4dfd2230860f7" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 40211 } } peers { permanent_uuid: "3bec1377543d433e832f77e9ebc6d740" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41097 } } peers { permanent_uuid: "668165b874ca4fcb8714fa15f91de91b" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41173 } } }
I20260926 00:59:44.335448 5612 tablet_replica.cc:442] T 8e0d65cda77944dfbf9acc18f4e9d041 P 5ea28b33245c4862a7c4dfd2230860f7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 00:59:44.335678 5611 tablet_replica.cc:439] T 8e0d65cda77944dfbf9acc18f4e9d041 P 668165b874ca4fcb8714fa15f91de91b: TxnStatusTablet state changed. Reason: New leader 3bec1377543d433e832f77e9ebc6d740. Latest consensus state: current_term: 1 leader_uuid: "3bec1377543d433e832f77e9ebc6d740" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea28b33245c4862a7c4dfd2230860f7" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 40211 } } peers { permanent_uuid: "3bec1377543d433e832f77e9ebc6d740" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41097 } } peers { permanent_uuid: "668165b874ca4fcb8714fa15f91de91b" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41173 } } }
I20260926 00:59:44.335791 5611 tablet_replica.cc:442] T 8e0d65cda77944dfbf9acc18f4e9d041 P 668165b874ca4fcb8714fa15f91de91b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 00:59:44.336474 5615 tablet_replica.cc:439] T 8e0d65cda77944dfbf9acc18f4e9d041 P 3bec1377543d433e832f77e9ebc6d740: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3bec1377543d433e832f77e9ebc6d740" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea28b33245c4862a7c4dfd2230860f7" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 40211 } } peers { permanent_uuid: "3bec1377543d433e832f77e9ebc6d740" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41097 } } peers { permanent_uuid: "668165b874ca4fcb8714fa15f91de91b" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41173 } } }
I20260926 00:59:44.336566 5615 tablet_replica.cc:442] T 8e0d65cda77944dfbf9acc18f4e9d041 P 3bec1377543d433e832f77e9ebc6d740: This TxnStatusTablet replica's current role is: LEADER
I20260926 00:59:44.336611 5630 mvcc.cc:205] Tried to move back new op lower bound from 7333414438229172224 to 7333414438195703808. Current Snapshot: MvccSnapshot[applied={T|T < 7333414438229172224}]
I20260926 00:59:44.337147 5612 tablet_replica.cc:439] T 8e0d65cda77944dfbf9acc18f4e9d041 P 5ea28b33245c4862a7c4dfd2230860f7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3bec1377543d433e832f77e9ebc6d740" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea28b33245c4862a7c4dfd2230860f7" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 40211 } } peers { permanent_uuid: "3bec1377543d433e832f77e9ebc6d740" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41097 } } peers { permanent_uuid: "668165b874ca4fcb8714fa15f91de91b" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41173 } } }
I20260926 00:59:44.337236 5612 tablet_replica.cc:442] T 8e0d65cda77944dfbf9acc18f4e9d041 P 5ea28b33245c4862a7c4dfd2230860f7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 00:59:44.337639 5631 mvcc.cc:205] Tried to move back new op lower bound from 7333414438229172224 to 7333414438195703808. Current Snapshot: MvccSnapshot[applied={T|T < 7333414438229172224}]
I20260926 00:59:44.338693 5613 tablet_replica.cc:439] T 8e0d65cda77944dfbf9acc18f4e9d041 P 3bec1377543d433e832f77e9ebc6d740: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3bec1377543d433e832f77e9ebc6d740" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea28b33245c4862a7c4dfd2230860f7" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 40211 } } peers { permanent_uuid: "3bec1377543d433e832f77e9ebc6d740" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41097 } } peers { permanent_uuid: "668165b874ca4fcb8714fa15f91de91b" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41173 } } }
I20260926 00:59:44.338764 5613 tablet_replica.cc:442] T 8e0d65cda77944dfbf9acc18f4e9d041 P 3bec1377543d433e832f77e9ebc6d740: This TxnStatusTablet replica's current role is: LEADER
I20260926 00:59:44.339278 5633 mvcc.cc:205] Tried to move back new op lower bound from 7333414438229172224 to 7333414438195703808. Current Snapshot: MvccSnapshot[applied={T|T < 7333414438229172224}]
I20260926 00:59:44.339586 5611 tablet_replica.cc:439] T 8e0d65cda77944dfbf9acc18f4e9d041 P 668165b874ca4fcb8714fa15f91de91b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3bec1377543d433e832f77e9ebc6d740" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea28b33245c4862a7c4dfd2230860f7" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 40211 } } peers { permanent_uuid: "3bec1377543d433e832f77e9ebc6d740" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41097 } } peers { permanent_uuid: "668165b874ca4fcb8714fa15f91de91b" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41173 } } }
I20260926 00:59:44.339665 5611 tablet_replica.cc:442] T 8e0d65cda77944dfbf9acc18f4e9d041 P 668165b874ca4fcb8714fa15f91de91b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 00:59:44.341166 5603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:44.338457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40588 (local address 127.3.75.2:41097)
0926 00:59:44.338489 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:44.338491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:44.338508 (+ 17us) server_negotiation.cc:407] Connection header received
0926 00:59:44.338528 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:44.338529 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:44.338538 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:44.338585 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 00:59:44.338719 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:44.339465 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:44.340565 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:44.340736 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:44.340804 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:44.340955 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 00:59:44.340977 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 00:59:44.341025 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:44.341089 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 00:59:44.341103 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 00:59:44.345580 5603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:44.343444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:46131 (local address 127.3.75.2:41097)
0926 00:59:44.343474 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:44.343476 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:44.343484 (+ 8us) server_negotiation.cc:407] Connection header received
0926 00:59:44.343547 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:44.343548 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:44.343556 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:44.343603 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 00:59:44.343729 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:44.344156 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:44.344894 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:44.345097 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:44.345178 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:44.345353 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 00:59:44.345380 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 00:59:44.345430 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:44.345492 (+ 62us) server_negotiation.cc:299] Negotiation successful
0926 00:59:44.345513 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":16}
I20260926 00:59:44.345582 5448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:44.343368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.2:41097 (local address 127.3.75.2:46131)
0926 00:59:44.343434 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:44.343445 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:44.343490 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:44.343604 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:44.343605 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:44.343606 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 00:59:44.343665 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:44.343666 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:44.344246 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:44.344247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:44.344799 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:44.344800 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:44.345178 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:44.345179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:44.345237 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:44.345252 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 00:59:44.345437 (+ 185us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 00:59:44.345439 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 00:59:44.345489 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 00:59:44.345523 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":5}
I20260926 00:59:44.347573 5256 raft_consensus.cc:1261] T f0642df07a124ae08ef4698214f12eed P 5ea28b33245c4862a7c4dfd2230860f7 [term 1 FOLLOWER]: Refusing update from remote peer 3bec1377543d433e832f77e9ebc6d740: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 00:59:44.347589 5536 raft_consensus.cc:1261] T f0642df07a124ae08ef4698214f12eed P 668165b874ca4fcb8714fa15f91de91b [term 1 FOLLOWER]: Refusing update from remote peer 3bec1377543d433e832f77e9ebc6d740: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 00:59:44.347791 5613 consensus_queue.cc:1035] T f0642df07a124ae08ef4698214f12eed P 3bec1377543d433e832f77e9ebc6d740 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ea28b33245c4862a7c4dfd2230860f7" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 40211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 00:59:44.347921 5641 consensus_queue.cc:1035] T f0642df07a124ae08ef4698214f12eed P 3bec1377543d433e832f77e9ebc6d740 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668165b874ca4fcb8714fa15f91de91b" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 00:59:44.358587 5098 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 00:59:44.358692 5098 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 00:59:44.359406 5098 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b5d54f69067e4c24a51d89252d078c71: Sending DeleteTablet for 3 replicas of tablet f0642df07a124ae08ef4698214f12eed
I20260926 00:59:44.359707 5516 tablet_service.cc:1514] Processing DeleteTablet for tablet f0642df07a124ae08ef4698214f12eed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 00:59:44 UTC) from {username='kudu', principal='kudu/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:54370
I20260926 00:59:44.359710 5236 tablet_service.cc:1514] Processing DeleteTablet for tablet f0642df07a124ae08ef4698214f12eed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 00:59:44 UTC) from {username='kudu', principal='kudu/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:56984
I20260926 00:59:44.360153 5645 tablet_replica.cc:330] T f0642df07a124ae08ef4698214f12eed P 5ea28b33245c4862a7c4dfd2230860f7: stopping tablet replica
I20260926 00:59:44.360178 5375 tablet_service.cc:1514] Processing DeleteTablet for tablet f0642df07a124ae08ef4698214f12eed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 00:59:44 UTC) from {username='kudu', principal='kudu/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:40564
I20260926 00:59:44.360236 5098 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58220:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 00:59:44.360399 5098 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 00:59:44.360433 5647 tablet_replica.cc:330] T f0642df07a124ae08ef4698214f12eed P 3bec1377543d433e832f77e9ebc6d740: stopping tablet replica
I20260926 00:59:44.360529 5647 raft_consensus.cc:2229] T f0642df07a124ae08ef4698214f12eed P 3bec1377543d433e832f77e9ebc6d740 [term 1 LEADER]: Raft consensus shutting down.
I20260926 00:59:44.360677 5646 tablet_replica.cc:330] T f0642df07a124ae08ef4698214f12eed P 668165b874ca4fcb8714fa15f91de91b: stopping tablet replica
I20260926 00:59:44.360728 5647 raft_consensus.cc:2258] T f0642df07a124ae08ef4698214f12eed P 3bec1377543d433e832f77e9ebc6d740 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 00:59:44.360814 5645 raft_consensus.cc:2229] T f0642df07a124ae08ef4698214f12eed P 5ea28b33245c4862a7c4dfd2230860f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 00:59:44.360880 5645 raft_consensus.cc:2258] T f0642df07a124ae08ef4698214f12eed P 5ea28b33245c4862a7c4dfd2230860f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 00:59:44.361233 5647 ts_tablet_manager.cc:1905] T f0642df07a124ae08ef4698214f12eed P 3bec1377543d433e832f77e9ebc6d740: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 00:59:44.361297 5645 ts_tablet_manager.cc:1905] T f0642df07a124ae08ef4698214f12eed P 5ea28b33245c4862a7c4dfd2230860f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 00:59:44.362246 5236 tablet_service.cc:1467] Processing CreateTablet for tablet ba6f18fd93e84b75af8121f6e21f5d7a (DEFAULT_TABLE table=test-table [id=dccf43bfa06044c8902d0de7be22c5bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 00:59:44.362401 5236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba6f18fd93e84b75af8121f6e21f5d7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 00:59:44.363095 5645 ts_tablet_manager.cc:1918] T f0642df07a124ae08ef4698214f12eed P 5ea28b33245c4862a7c4dfd2230860f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 00:59:44.363112 5647 ts_tablet_manager.cc:1918] T f0642df07a124ae08ef4698214f12eed P 3bec1377543d433e832f77e9ebc6d740: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 00:59:44.363153 5645 log.cc:1196] T f0642df07a124ae08ef4698214f12eed P 5ea28b33245c4862a7c4dfd2230860f7: Deleting WAL directory at /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/ts-0/wal/wals/f0642df07a124ae08ef4698214f12eed
I20260926 00:59:44.363180 5647 log.cc:1196] T f0642df07a124ae08ef4698214f12eed P 3bec1377543d433e832f77e9ebc6d740: Deleting WAL directory at /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/ts-1/wal/wals/f0642df07a124ae08ef4698214f12eed
I20260926 00:59:44.363401 5645 ts_tablet_manager.cc:1939] T f0642df07a124ae08ef4698214f12eed P 5ea28b33245c4862a7c4dfd2230860f7: Deleting consensus metadata
I20260926 00:59:44.363435 5647 ts_tablet_manager.cc:1939] T f0642df07a124ae08ef4698214f12eed P 3bec1377543d433e832f77e9ebc6d740: Deleting consensus metadata
I20260926 00:59:44.363692 5375 tablet_service.cc:1467] Processing CreateTablet for tablet ba6f18fd93e84b75af8121f6e21f5d7a (DEFAULT_TABLE table=test-table [id=dccf43bfa06044c8902d0de7be22c5bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 00:59:44.363801 5375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba6f18fd93e84b75af8121f6e21f5d7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 00:59:44.363780 5085 catalog_manager.cc:4841] TS 3bec1377543d433e832f77e9ebc6d740 (127.3.75.2:41097): tablet f0642df07a124ae08ef4698214f12eed (table test-table [id=b3ba97112ab04f3da84d80fe9e605e46]) successfully deleted
I20260926 00:59:44.364710 5609 tablet_bootstrap.cc:492] T ba6f18fd93e84b75af8121f6e21f5d7a P 3bec1377543d433e832f77e9ebc6d740: Bootstrap starting.
I20260926 00:59:44.364742 5606 tablet_bootstrap.cc:492] T ba6f18fd93e84b75af8121f6e21f5d7a P 5ea28b33245c4862a7c4dfd2230860f7: Bootstrap starting.
I20260926 00:59:44.365074 5606 tablet_bootstrap.cc:654] T ba6f18fd93e84b75af8121f6e21f5d7a P 5ea28b33245c4862a7c4dfd2230860f7: Neither blocks nor log segments found. Creating new log.
I20260926 00:59:44.365090 5609 tablet_bootstrap.cc:654] T ba6f18fd93e84b75af8121f6e21f5d7a P 3bec1377543d433e832f77e9ebc6d740: Neither blocks nor log segments found. Creating new log.
I20260926 00:59:44.365450 5085 catalog_manager.cc:4841] TS 5ea28b33245c4862a7c4dfd2230860f7 (127.3.75.1:40211): tablet f0642df07a124ae08ef4698214f12eed (table test-table [id=b3ba97112ab04f3da84d80fe9e605e46]) successfully deleted
I20260926 00:59:44.365589 5609 tablet_bootstrap.cc:492] T ba6f18fd93e84b75af8121f6e21f5d7a P 3bec1377543d433e832f77e9ebc6d740: No bootstrap required, opened a new log
I20260926 00:59:44.365634 5609 ts_tablet_manager.cc:1397] T ba6f18fd93e84b75af8121f6e21f5d7a P 3bec1377543d433e832f77e9ebc6d740: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 00:59:44.365736 5606 tablet_bootstrap.cc:492] T ba6f18fd93e84b75af8121f6e21f5d7a P 5ea28b33245c4862a7c4dfd2230860f7: No bootstrap required, opened a new log
I20260926 00:59:44.365782 5606 ts_tablet_manager.cc:1397] T ba6f18fd93e84b75af8121f6e21f5d7a P 5ea28b33245c4862a7c4dfd2230860f7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 00:59:44.365761 5609 raft_consensus.cc:348] T ba6f18fd93e84b75af8121f6e21f5d7a P 3bec1377543d433e832f77e9ebc6d740 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea28b33245c4862a7c4dfd2230860f7" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 40211 } } peers { permanent_uuid: "3bec1377543d433e832f77e9ebc6d740" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41097 } } peers { permanent_uuid: "668165b874ca4fcb8714fa15f91de91b" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41173 } }
I20260926 00:59:44.365816 5609 raft_consensus.cc:374] T ba6f18fd93e84b75af8121f6e21f5d7a P 3bec1377543d433e832f77e9ebc6d740 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 00:59:44.365842 5609 raft_consensus.cc:729] T ba6f18fd93e84b75af8121f6e21f5d7a P 3bec1377543d433e832f77e9ebc6d740 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bec1377543d433e832f77e9ebc6d740, State: Initialized, Role: FOLLOWER
I20260926 00:59:44.365880 5609 consensus_queue.cc:260] T ba6f18fd93e84b75af8121f6e21f5d7a P 3bec1377543d433e832f77e9ebc6d740 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea28b33245c4862a7c4dfd2230860f7" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 40211 } } peers { permanent_uuid: "3bec1377543d433e832f77e9ebc6d740" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41097 } } peers { permanent_uuid: "668165b874ca4fcb8714fa15f91de91b" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41173 } }
I20260926 00:59:44.365913 5606 raft_consensus.cc:348] T ba6f18fd93e84b75af8121f6e21f5d7a P 5ea28b33245c4862a7c4dfd2230860f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea28b33245c4862a7c4dfd2230860f7" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 40211 } } peers { permanent_uuid: "3bec1377543d433e832f77e9ebc6d740" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41097 } } peers { permanent_uuid: "668165b874ca4fcb8714fa15f91de91b" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41173 } }
I20260926 00:59:44.365978 5606 raft_consensus.cc:374] T ba6f18fd93e84b75af8121f6e21f5d7a P 5ea28b33245c4862a7c4dfd2230860f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 00:59:44.365964 5609 ts_tablet_manager.cc:1428] T ba6f18fd93e84b75af8121f6e21f5d7a P 3bec1377543d433e832f77e9ebc6d740: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:44.366003 5606 raft_consensus.cc:729] T ba6f18fd93e84b75af8121f6e21f5d7a P 5ea28b33245c4862a7c4dfd2230860f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ea28b33245c4862a7c4dfd2230860f7, State: Initialized, Role: FOLLOWER
I20260926 00:59:44.366043 5606 consensus_queue.cc:260] T ba6f18fd93e84b75af8121f6e21f5d7a P 5ea28b33245c4862a7c4dfd2230860f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea28b33245c4862a7c4dfd2230860f7" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 40211 } } peers { permanent_uuid: "3bec1377543d433e832f77e9ebc6d740" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41097 } } peers { permanent_uuid: "668165b874ca4fcb8714fa15f91de91b" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41173 } }
I20260926 00:59:44.366133 5606 ts_tablet_manager.cc:1428] T ba6f18fd93e84b75af8121f6e21f5d7a P 5ea28b33245c4862a7c4dfd2230860f7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:44.361968 5516 tablet_service.cc:1467] Processing CreateTablet for tablet ba6f18fd93e84b75af8121f6e21f5d7a (DEFAULT_TABLE table=test-table [id=dccf43bfa06044c8902d0de7be22c5bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 00:59:44.366294 5516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba6f18fd93e84b75af8121f6e21f5d7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 00:59:44.367055 5646 raft_consensus.cc:2229] T f0642df07a124ae08ef4698214f12eed P 668165b874ca4fcb8714fa15f91de91b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 00:59:44.367151 5646 raft_consensus.cc:2258] T f0642df07a124ae08ef4698214f12eed P 668165b874ca4fcb8714fa15f91de91b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 00:59:44.367625 5605 tablet_bootstrap.cc:492] T ba6f18fd93e84b75af8121f6e21f5d7a P 668165b874ca4fcb8714fa15f91de91b: Bootstrap starting.
I20260926 00:59:44.367942 5646 ts_tablet_manager.cc:1905] T f0642df07a124ae08ef4698214f12eed P 668165b874ca4fcb8714fa15f91de91b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 00:59:44.368005 5605 tablet_bootstrap.cc:654] T ba6f18fd93e84b75af8121f6e21f5d7a P 668165b874ca4fcb8714fa15f91de91b: Neither blocks nor log segments found. Creating new log.
I20260926 00:59:44.368616 5605 tablet_bootstrap.cc:492] T ba6f18fd93e84b75af8121f6e21f5d7a P 668165b874ca4fcb8714fa15f91de91b: No bootstrap required, opened a new log
I20260926 00:59:44.368667 5605 ts_tablet_manager.cc:1397] T ba6f18fd93e84b75af8121f6e21f5d7a P 668165b874ca4fcb8714fa15f91de91b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 00:59:44.368788 5605 raft_consensus.cc:348] T ba6f18fd93e84b75af8121f6e21f5d7a P 668165b874ca4fcb8714fa15f91de91b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea28b33245c4862a7c4dfd2230860f7" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 40211 } } peers { permanent_uuid: "3bec1377543d433e832f77e9ebc6d740" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41097 } } peers { permanent_uuid: "668165b874ca4fcb8714fa15f91de91b" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41173 } }
I20260926 00:59:44.368852 5605 raft_consensus.cc:374] T ba6f18fd93e84b75af8121f6e21f5d7a P 668165b874ca4fcb8714fa15f91de91b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 00:59:44.368872 5605 raft_consensus.cc:729] T ba6f18fd93e84b75af8121f6e21f5d7a P 668165b874ca4fcb8714fa15f91de91b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668165b874ca4fcb8714fa15f91de91b, State: Initialized, Role: FOLLOWER
I20260926 00:59:44.368916 5605 consensus_queue.cc:260] T ba6f18fd93e84b75af8121f6e21f5d7a P 668165b874ca4fcb8714fa15f91de91b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea28b33245c4862a7c4dfd2230860f7" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 40211 } } peers { permanent_uuid: "3bec1377543d433e832f77e9ebc6d740" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41097 } } peers { permanent_uuid: "668165b874ca4fcb8714fa15f91de91b" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41173 } }
I20260926 00:59:44.369019 5605 ts_tablet_manager.cc:1428] T ba6f18fd93e84b75af8121f6e21f5d7a P 668165b874ca4fcb8714fa15f91de91b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:44.369513 5646 ts_tablet_manager.cc:1918] T f0642df07a124ae08ef4698214f12eed P 668165b874ca4fcb8714fa15f91de91b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 00:59:44.369598 5646 log.cc:1196] T f0642df07a124ae08ef4698214f12eed P 668165b874ca4fcb8714fa15f91de91b: Deleting WAL directory at /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/ts-2/wal/wals/f0642df07a124ae08ef4698214f12eed
I20260926 00:59:44.369869 5646 ts_tablet_manager.cc:1939] T f0642df07a124ae08ef4698214f12eed P 668165b874ca4fcb8714fa15f91de91b: Deleting consensus metadata
I20260926 00:59:44.370221 5084 catalog_manager.cc:4841] TS 668165b874ca4fcb8714fa15f91de91b (127.3.75.3:41173): tablet f0642df07a124ae08ef4698214f12eed (table test-table [id=b3ba97112ab04f3da84d80fe9e605e46]) successfully deleted
I20260926 00:59:44.380548 5612 raft_consensus.cc:482] T ba6f18fd93e84b75af8121f6e21f5d7a P 5ea28b33245c4862a7c4dfd2230860f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 00:59:44.380630 5612 raft_consensus.cc:504] T ba6f18fd93e84b75af8121f6e21f5d7a P 5ea28b33245c4862a7c4dfd2230860f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea28b33245c4862a7c4dfd2230860f7" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 40211 } } peers { permanent_uuid: "3bec1377543d433e832f77e9ebc6d740" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41097 } } peers { permanent_uuid: "668165b874ca4fcb8714fa15f91de91b" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41173 } }
I20260926 00:59:44.380918 5612 leader_election.cc:288] T ba6f18fd93e84b75af8121f6e21f5d7a P 5ea28b33245c4862a7c4dfd2230860f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3bec1377543d433e832f77e9ebc6d740 (127.3.75.2:41097), 668165b874ca4fcb8714fa15f91de91b (127.3.75.3:41173)
I20260926 00:59:44.384001 5603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:44.381214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:34567 (local address 127.3.75.2:41097)
0926 00:59:44.381278 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:44.381281 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:44.381343 (+ 62us) server_negotiation.cc:407] Connection header received
0926 00:59:44.381419 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:44.381421 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:44.381431 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:44.381478 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 00:59:44.381646 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:44.382101 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:44.383312 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:44.383651 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:44.383728 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:44.383739 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:44.383904 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 00:59:44.383942 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 00:59:44.384034 5652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:44.381066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.2:41097 (local address 127.3.75.1:34567)
0926 00:59:44.381281 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:44.381292 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:44.381358 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:44.381494 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:44.381495 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:44.381499 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 00:59:44.381561 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:44.381563 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:44.382183 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:44.382184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:44.383229 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:44.383232 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:44.383739 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:44.383740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:44.383824 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:44.383828 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 00:59:44.383886 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 00:59:44.383937 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":10,"thread_start_us":59,"threads_started":1}
I20260926 00:59:44.384318 5396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba6f18fd93e84b75af8121f6e21f5d7a" candidate_uuid: "5ea28b33245c4862a7c4dfd2230860f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bec1377543d433e832f77e9ebc6d740" is_pre_election: true
I20260926 00:59:44.384442 5396 raft_consensus.cc:2454] T ba6f18fd93e84b75af8121f6e21f5d7a P 3bec1377543d433e832f77e9ebc6d740 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ea28b33245c4862a7c4dfd2230860f7 in term 0.
I20260926 00:59:44.384878 5601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:44.381227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:35989 (local address 127.3.75.3:41173)
0926 00:59:44.381311 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:44.381313 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:44.381365 (+ 52us) server_negotiation.cc:407] Connection header received
0926 00:59:44.381473 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:44.381474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:44.381484 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:44.381533 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 00:59:44.381727 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:44.382496 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:44.383924 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:44.384507 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:44.384555 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:44.384564 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:44.384785 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 00:59:44.384818 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 00:59:44.385089 5651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:44.381113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.3:41173 (local address 127.3.75.1:35989)
0926 00:59:44.381303 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:44.381314 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:44.381389 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:44.381576 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:44.381577 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:44.381579 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 00:59:44.381653 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:44.381655 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:44.382676 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:44.382677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:44.383820 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:44.383823 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:44.384630 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:44.384631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:44.384706 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:44.384709 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 00:59:44.384944 (+ 235us) client_negotiation.cc:240] Negotiation successful
0926 00:59:44.384996 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":74,"threads_started":1}
I20260926 00:59:44.385424 5189 leader_election.cc:302] T ba6f18fd93e84b75af8121f6e21f5d7a P 5ea28b33245c4862a7c4dfd2230860f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bec1377543d433e832f77e9ebc6d740, 5ea28b33245c4862a7c4dfd2230860f7; no voters:
I20260926 00:59:44.385499 5536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba6f18fd93e84b75af8121f6e21f5d7a" candidate_uuid: "5ea28b33245c4862a7c4dfd2230860f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668165b874ca4fcb8714fa15f91de91b" is_pre_election: true
I20260926 00:59:44.385550 5612 raft_consensus.cc:2781] T ba6f18fd93e84b75af8121f6e21f5d7a P 5ea28b33245c4862a7c4dfd2230860f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 00:59:44.385578 5536 raft_consensus.cc:2454] T ba6f18fd93e84b75af8121f6e21f5d7a P 668165b874ca4fcb8714fa15f91de91b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ea28b33245c4862a7c4dfd2230860f7 in term 0.
I20260926 00:59:44.385586 5612 raft_consensus.cc:482] T ba6f18fd93e84b75af8121f6e21f5d7a P 5ea28b33245c4862a7c4dfd2230860f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 00:59:44.385609 5612 raft_consensus.cc:3037] T ba6f18fd93e84b75af8121f6e21f5d7a P 5ea28b33245c4862a7c4dfd2230860f7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 00:59:44.386263 5612 raft_consensus.cc:504] T ba6f18fd93e84b75af8121f6e21f5d7a P 5ea28b33245c4862a7c4dfd2230860f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea28b33245c4862a7c4dfd2230860f7" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 40211 } } peers { permanent_uuid: "3bec1377543d433e832f77e9ebc6d740" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41097 } } peers { permanent_uuid: "668165b874ca4fcb8714fa15f91de91b" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41173 } }
I20260926 00:59:44.386404 5612 leader_election.cc:288] T ba6f18fd93e84b75af8121f6e21f5d7a P 5ea28b33245c4862a7c4dfd2230860f7 [CANDIDATE]: Term 1 election: Requested vote from peers 3bec1377543d433e832f77e9ebc6d740 (127.3.75.2:41097), 668165b874ca4fcb8714fa15f91de91b (127.3.75.3:41173)
I20260926 00:59:44.386586 5536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba6f18fd93e84b75af8121f6e21f5d7a" candidate_uuid: "5ea28b33245c4862a7c4dfd2230860f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668165b874ca4fcb8714fa15f91de91b"
I20260926 00:59:44.386658 5536 raft_consensus.cc:3037] T ba6f18fd93e84b75af8121f6e21f5d7a P 668165b874ca4fcb8714fa15f91de91b [term 0 FOLLOWER]: Advancing to term 1
I20260926 00:59:44.386693 5396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba6f18fd93e84b75af8121f6e21f5d7a" candidate_uuid: "5ea28b33245c4862a7c4dfd2230860f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bec1377543d433e832f77e9ebc6d740"
I20260926 00:59:44.386786 5396 raft_consensus.cc:3037] T ba6f18fd93e84b75af8121f6e21f5d7a P 3bec1377543d433e832f77e9ebc6d740 [term 0 FOLLOWER]: Advancing to term 1
I20260926 00:59:44.387349 5536 raft_consensus.cc:2454] T ba6f18fd93e84b75af8121f6e21f5d7a P 668165b874ca4fcb8714fa15f91de91b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ea28b33245c4862a7c4dfd2230860f7 in term 1.
I20260926 00:59:44.387523 5396 raft_consensus.cc:2454] T ba6f18fd93e84b75af8121f6e21f5d7a P 3bec1377543d433e832f77e9ebc6d740 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ea28b33245c4862a7c4dfd2230860f7 in term 1.
I20260926 00:59:44.387684 5188 leader_election.cc:302] T ba6f18fd93e84b75af8121f6e21f5d7a P 5ea28b33245c4862a7c4dfd2230860f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ea28b33245c4862a7c4dfd2230860f7, 668165b874ca4fcb8714fa15f91de91b; no voters:
I20260926 00:59:44.387797 5612 raft_consensus.cc:2781] T ba6f18fd93e84b75af8121f6e21f5d7a P 5ea28b33245c4862a7c4dfd2230860f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 00:59:44.387950 5612 raft_consensus.cc:686] T ba6f18fd93e84b75af8121f6e21f5d7a P 5ea28b33245c4862a7c4dfd2230860f7 [term 1 LEADER]: Becoming Leader. State: Replica: 5ea28b33245c4862a7c4dfd2230860f7, State: Running, Role: LEADER
I20260926 00:59:44.388026 5612 consensus_queue.cc:237] T ba6f18fd93e84b75af8121f6e21f5d7a P 5ea28b33245c4862a7c4dfd2230860f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea28b33245c4862a7c4dfd2230860f7" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 40211 } } peers { permanent_uuid: "3bec1377543d433e832f77e9ebc6d740" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41097 } } peers { permanent_uuid: "668165b874ca4fcb8714fa15f91de91b" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41173 } }
I20260926 00:59:44.388585 5098 catalog_manager.cc:5495] T ba6f18fd93e84b75af8121f6e21f5d7a P 5ea28b33245c4862a7c4dfd2230860f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ea28b33245c4862a7c4dfd2230860f7 (127.3.75.1). New cstate: current_term: 1 leader_uuid: "5ea28b33245c4862a7c4dfd2230860f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea28b33245c4862a7c4dfd2230860f7" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 40211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3bec1377543d433e832f77e9ebc6d740" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668165b874ca4fcb8714fa15f91de91b" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41173 } health_report { overall_health: UNKNOWN } } }
I20260926 00:59:44.399574 5604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:44.397125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56996 (local address 127.3.75.1:40211)
0926 00:59:44.397164 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:44.397166 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:44.397174 (+ 8us) server_negotiation.cc:407] Connection header received
0926 00:59:44.397237 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:44.397238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:44.397254 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:44.397320 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 00:59:44.397470 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:44.397993 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:44.398719 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:44.398991 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:44.399081 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:44.399246 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 00:59:44.399352 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 00:59:44.399431 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:44.399472 (+ 41us) server_negotiation.cc:299] Negotiation successful
0926 00:59:44.399492 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 00:59:44.404701 5603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:44.401996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:44247 (local address 127.3.75.2:41097)
0926 00:59:44.402040 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:44.402042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:44.402184 (+ 142us) server_negotiation.cc:407] Connection header received
0926 00:59:44.402243 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:44.402244 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:44.402252 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:44.402297 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 00:59:44.402454 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:44.402923 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:44.403991 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:44.404212 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:44.404264 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:44.404461 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 00:59:44.404484 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 00:59:44.404527 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:44.404621 (+ 94us) server_negotiation.cc:299] Negotiation successful
0926 00:59:44.404640 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 00:59:44.404829 5657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:44.401904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.2:41097 (local address 127.3.75.1:44247)
0926 00:59:44.402108 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:44.402120 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:44.402186 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:44.402317 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:44.402317 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:44.402319 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 00:59:44.402391 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:44.402392 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:44.403005 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:44.403006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:44.403902 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:44.403905 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:44.404293 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:44.404294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:44.404362 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:44.404379 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 00:59:44.404552 (+ 173us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 00:59:44.404555 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 00:59:44.404613 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 00:59:44.404657 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":119,"threads_started":1}
I20260926 00:59:44.406759 5536 raft_consensus.cc:1261] T ba6f18fd93e84b75af8121f6e21f5d7a P 668165b874ca4fcb8714fa15f91de91b [term 1 FOLLOWER]: Refusing update from remote peer 5ea28b33245c4862a7c4dfd2230860f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 00:59:44.406839 5396 raft_consensus.cc:1261] T ba6f18fd93e84b75af8121f6e21f5d7a P 3bec1377543d433e832f77e9ebc6d740 [term 1 FOLLOWER]: Refusing update from remote peer 5ea28b33245c4862a7c4dfd2230860f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 00:59:44.407042 5612 consensus_queue.cc:1035] T ba6f18fd93e84b75af8121f6e21f5d7a P 5ea28b33245c4862a7c4dfd2230860f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668165b874ca4fcb8714fa15f91de91b" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 00:59:44.407150 5653 consensus_queue.cc:1035] T ba6f18fd93e84b75af8121f6e21f5d7a P 5ea28b33245c4862a7c4dfd2230860f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bec1377543d433e832f77e9ebc6d740" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 00:59:44.415112 5604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:44.412464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:52799 (local address 127.3.75.1:40211)
0926 00:59:44.412630 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:44.412632 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:44.412639 (+ 7us) server_negotiation.cc:407] Connection header received
0926 00:59:44.412657 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:44.412657 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:44.412668 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:44.412741 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 00:59:44.412877 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:44.413549 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:44.414421 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:44.414688 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:44.414742 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:44.414894 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 00:59:44.414923 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 00:59:44.414975 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:44.415033 (+ 58us) server_negotiation.cc:299] Negotiation successful
0926 00:59:44.415050 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147}
I20260926 00:59:44.415127 5448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:44.412422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.1:40211 (local address 127.3.75.2:52799)
0926 00:59:44.412466 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:44.412476 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:44.412510 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:44.412760 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:44.412761 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:44.412762 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 00:59:44.412800 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:44.412801 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:44.413635 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:44.413635 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:44.414327 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:44.414330 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:44.414751 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:44.414752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:44.414801 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:44.414805 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 00:59:44.414978 (+ 173us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 00:59:44.414980 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 00:59:44.415029 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 00:59:44.415066 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
W20260926 00:59:44.415421 5631 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 00:59:45.939664 5098 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 00:59:45.939764 5098 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 00:59:45.940554 5098 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b5d54f69067e4c24a51d89252d078c71: Sending DeleteTablet for 3 replicas of tablet ba6f18fd93e84b75af8121f6e21f5d7a
I20260926 00:59:45.940865 5516 tablet_service.cc:1514] Processing DeleteTablet for tablet ba6f18fd93e84b75af8121f6e21f5d7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 00:59:45 UTC) from {username='kudu', principal='kudu/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:54370
I20260926 00:59:45.940886 5236 tablet_service.cc:1514] Processing DeleteTablet for tablet ba6f18fd93e84b75af8121f6e21f5d7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 00:59:45 UTC) from {username='kudu', principal='kudu/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:56984
I20260926 00:59:45.940884 5375 tablet_service.cc:1514] Processing DeleteTablet for tablet ba6f18fd93e84b75af8121f6e21f5d7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 00:59:45 UTC) from {username='kudu', principal='kudu/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:40564
I20260926 00:59:45.941136 5684 tablet_replica.cc:330] T ba6f18fd93e84b75af8121f6e21f5d7a P 3bec1377543d433e832f77e9ebc6d740: stopping tablet replica
I20260926 00:59:45.941146 5683 tablet_replica.cc:330] T ba6f18fd93e84b75af8121f6e21f5d7a P 668165b874ca4fcb8714fa15f91de91b: stopping tablet replica
I20260926 00:59:45.941203 5684 raft_consensus.cc:2229] T ba6f18fd93e84b75af8121f6e21f5d7a P 3bec1377543d433e832f77e9ebc6d740 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 00:59:45.941251 5684 raft_consensus.cc:2258] T ba6f18fd93e84b75af8121f6e21f5d7a P 3bec1377543d433e832f77e9ebc6d740 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 00:59:45.941272 5683 raft_consensus.cc:2229] T ba6f18fd93e84b75af8121f6e21f5d7a P 668165b874ca4fcb8714fa15f91de91b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 00:59:45.941316 5683 raft_consensus.cc:2258] T ba6f18fd93e84b75af8121f6e21f5d7a P 668165b874ca4fcb8714fa15f91de91b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 00:59:45.941527 5684 ts_tablet_manager.cc:1905] T ba6f18fd93e84b75af8121f6e21f5d7a P 3bec1377543d433e832f77e9ebc6d740: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 00:59:45.941560 5683 ts_tablet_manager.cc:1905] T ba6f18fd93e84b75af8121f6e21f5d7a P 668165b874ca4fcb8714fa15f91de91b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 00:59:45.942993 5685 tablet_replica.cc:330] T ba6f18fd93e84b75af8121f6e21f5d7a P 5ea28b33245c4862a7c4dfd2230860f7: stopping tablet replica
I20260926 00:59:45.943065 5685 raft_consensus.cc:2229] T ba6f18fd93e84b75af8121f6e21f5d7a P 5ea28b33245c4862a7c4dfd2230860f7 [term 1 LEADER]: Raft consensus shutting down.
I20260926 00:59:45.943162 5685 raft_consensus.cc:2258] T ba6f18fd93e84b75af8121f6e21f5d7a P 5ea28b33245c4862a7c4dfd2230860f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 00:59:45.943467 5685 ts_tablet_manager.cc:1905] T ba6f18fd93e84b75af8121f6e21f5d7a P 5ea28b33245c4862a7c4dfd2230860f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 00:59:45.943533 5683 ts_tablet_manager.cc:1918] T ba6f18fd93e84b75af8121f6e21f5d7a P 668165b874ca4fcb8714fa15f91de91b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 00:59:45.943604 5683 log.cc:1196] T ba6f18fd93e84b75af8121f6e21f5d7a P 668165b874ca4fcb8714fa15f91de91b: Deleting WAL directory at /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/ts-2/wal/wals/ba6f18fd93e84b75af8121f6e21f5d7a
I20260926 00:59:45.943876 5683 ts_tablet_manager.cc:1939] T ba6f18fd93e84b75af8121f6e21f5d7a P 668165b874ca4fcb8714fa15f91de91b: Deleting consensus metadata
I20260926 00:59:45.944180 5084 catalog_manager.cc:4841] TS 668165b874ca4fcb8714fa15f91de91b (127.3.75.3:41173): tablet ba6f18fd93e84b75af8121f6e21f5d7a (table test-table [id=dccf43bfa06044c8902d0de7be22c5bd]) successfully deleted
I20260926 00:59:45.944252 5684 ts_tablet_manager.cc:1918] T ba6f18fd93e84b75af8121f6e21f5d7a P 3bec1377543d433e832f77e9ebc6d740: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 00:59:45.944312 5684 log.cc:1196] T ba6f18fd93e84b75af8121f6e21f5d7a P 3bec1377543d433e832f77e9ebc6d740: Deleting WAL directory at /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/ts-1/wal/wals/ba6f18fd93e84b75af8121f6e21f5d7a
I20260926 00:59:45.944522 5684 ts_tablet_manager.cc:1939] T ba6f18fd93e84b75af8121f6e21f5d7a P 3bec1377543d433e832f77e9ebc6d740: Deleting consensus metadata
I20260926 00:59:45.944842 5085 catalog_manager.cc:4841] TS 3bec1377543d433e832f77e9ebc6d740 (127.3.75.2:41097): tablet ba6f18fd93e84b75af8121f6e21f5d7a (table test-table [id=dccf43bfa06044c8902d0de7be22c5bd]) successfully deleted
I20260926 00:59:45.945246 5685 ts_tablet_manager.cc:1918] T ba6f18fd93e84b75af8121f6e21f5d7a P 5ea28b33245c4862a7c4dfd2230860f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 00:59:45.945320 5685 log.cc:1196] T ba6f18fd93e84b75af8121f6e21f5d7a P 5ea28b33245c4862a7c4dfd2230860f7: Deleting WAL directory at /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/ts-0/wal/wals/ba6f18fd93e84b75af8121f6e21f5d7a
I20260926 00:59:45.945544 5685 ts_tablet_manager.cc:1939] T ba6f18fd93e84b75af8121f6e21f5d7a P 5ea28b33245c4862a7c4dfd2230860f7: Deleting consensus metadata
I20260926 00:59:45.945897 5085 catalog_manager.cc:4841] TS 5ea28b33245c4862a7c4dfd2230860f7 (127.3.75.1:40211): tablet ba6f18fd93e84b75af8121f6e21f5d7a (table test-table [id=dccf43bfa06044c8902d0de7be22c5bd]) successfully deleted
Sep 26 00:59:45 dist-test-slave-7v10 krb5kdc[5054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384385, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 00:59:45 dist-test-slave-7v10 krb5kdc[5054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384385, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 00:59:45.964798 5693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:45.957598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58236 (local address 127.3.75.62:33135)
0926 00:59:45.957743 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:45.957745 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:45.957770 (+ 25us) server_negotiation.cc:407] Connection header received
0926 00:59:45.957846 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:45.957847 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:45.957859 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:45.957942 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:45.958486 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:45.959061 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:45.959806 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:45.960023 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:45.960106 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:45.962240 (+ 2134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:45.962241 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:45.962242 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 00:59:45.962248 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:45.963824 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:45.964142 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:45.964143 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:45.964144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:45.964200 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:45.964366 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:45.964367 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:45.964367 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:45.964490 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:45.964565 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:45.964670 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 00:59:45.964733 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":73,"threads_started":1}
I20260926 00:59:45.966255 5098 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58236:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 00:59:45.966419 5098 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 00:59:45.967890 5375 tablet_service.cc:1467] Processing CreateTablet for tablet 55f09097eaad47b582fb9d5302849049 (DEFAULT_TABLE table=test-table [id=7eda5a747b13430e8d797abf70614484]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 00:59:45.967876 5236 tablet_service.cc:1467] Processing CreateTablet for tablet 55f09097eaad47b582fb9d5302849049 (DEFAULT_TABLE table=test-table [id=7eda5a747b13430e8d797abf70614484]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 00:59:45.967901 5516 tablet_service.cc:1467] Processing CreateTablet for tablet 55f09097eaad47b582fb9d5302849049 (DEFAULT_TABLE table=test-table [id=7eda5a747b13430e8d797abf70614484]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 00:59:45.968015 5375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55f09097eaad47b582fb9d5302849049. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 00:59:45.968014 5236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55f09097eaad47b582fb9d5302849049. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 00:59:45.968048 5516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55f09097eaad47b582fb9d5302849049. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 00:59:45.969455 5694 tablet_bootstrap.cc:492] T 55f09097eaad47b582fb9d5302849049 P 668165b874ca4fcb8714fa15f91de91b: Bootstrap starting.
I20260926 00:59:45.969535 5695 tablet_bootstrap.cc:492] T 55f09097eaad47b582fb9d5302849049 P 5ea28b33245c4862a7c4dfd2230860f7: Bootstrap starting.
I20260926 00:59:45.969583 5696 tablet_bootstrap.cc:492] T 55f09097eaad47b582fb9d5302849049 P 3bec1377543d433e832f77e9ebc6d740: Bootstrap starting.
I20260926 00:59:45.969848 5694 tablet_bootstrap.cc:654] T 55f09097eaad47b582fb9d5302849049 P 668165b874ca4fcb8714fa15f91de91b: Neither blocks nor log segments found. Creating new log.
I20260926 00:59:45.970002 5695 tablet_bootstrap.cc:654] T 55f09097eaad47b582fb9d5302849049 P 5ea28b33245c4862a7c4dfd2230860f7: Neither blocks nor log segments found. Creating new log.
I20260926 00:59:45.970003 5696 tablet_bootstrap.cc:654] T 55f09097eaad47b582fb9d5302849049 P 3bec1377543d433e832f77e9ebc6d740: Neither blocks nor log segments found. Creating new log.
I20260926 00:59:45.970479 5694 tablet_bootstrap.cc:492] T 55f09097eaad47b582fb9d5302849049 P 668165b874ca4fcb8714fa15f91de91b: No bootstrap required, opened a new log
I20260926 00:59:45.970551 5694 ts_tablet_manager.cc:1397] T 55f09097eaad47b582fb9d5302849049 P 668165b874ca4fcb8714fa15f91de91b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 00:59:45.970605 5696 tablet_bootstrap.cc:492] T 55f09097eaad47b582fb9d5302849049 P 3bec1377543d433e832f77e9ebc6d740: No bootstrap required, opened a new log
I20260926 00:59:45.970664 5696 ts_tablet_manager.cc:1397] T 55f09097eaad47b582fb9d5302849049 P 3bec1377543d433e832f77e9ebc6d740: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 00:59:45.970752 5695 tablet_bootstrap.cc:492] T 55f09097eaad47b582fb9d5302849049 P 5ea28b33245c4862a7c4dfd2230860f7: No bootstrap required, opened a new log
I20260926 00:59:45.970713 5694 raft_consensus.cc:348] T 55f09097eaad47b582fb9d5302849049 P 668165b874ca4fcb8714fa15f91de91b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea28b33245c4862a7c4dfd2230860f7" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 40211 } } peers { permanent_uuid: "3bec1377543d433e832f77e9ebc6d740" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41097 } } peers { permanent_uuid: "668165b874ca4fcb8714fa15f91de91b" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41173 } }
I20260926 00:59:45.970785 5694 raft_consensus.cc:374] T 55f09097eaad47b582fb9d5302849049 P 668165b874ca4fcb8714fa15f91de91b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 00:59:45.970798 5695 ts_tablet_manager.cc:1397] T 55f09097eaad47b582fb9d5302849049 P 5ea28b33245c4862a7c4dfd2230860f7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 00:59:45.970804 5694 raft_consensus.cc:729] T 55f09097eaad47b582fb9d5302849049 P 668165b874ca4fcb8714fa15f91de91b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668165b874ca4fcb8714fa15f91de91b, State: Initialized, Role: FOLLOWER
I20260926 00:59:45.970779 5696 raft_consensus.cc:348] T 55f09097eaad47b582fb9d5302849049 P 3bec1377543d433e832f77e9ebc6d740 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea28b33245c4862a7c4dfd2230860f7" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 40211 } } peers { permanent_uuid: "3bec1377543d433e832f77e9ebc6d740" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41097 } } peers { permanent_uuid: "668165b874ca4fcb8714fa15f91de91b" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41173 } }
I20260926 00:59:45.970846 5696 raft_consensus.cc:374] T 55f09097eaad47b582fb9d5302849049 P 3bec1377543d433e832f77e9ebc6d740 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 00:59:45.970893 5696 raft_consensus.cc:729] T 55f09097eaad47b582fb9d5302849049 P 3bec1377543d433e832f77e9ebc6d740 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bec1377543d433e832f77e9ebc6d740, State: Initialized, Role: FOLLOWER
I20260926 00:59:45.970851 5694 consensus_queue.cc:260] T 55f09097eaad47b582fb9d5302849049 P 668165b874ca4fcb8714fa15f91de91b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea28b33245c4862a7c4dfd2230860f7" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 40211 } } peers { permanent_uuid: "3bec1377543d433e832f77e9ebc6d740" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41097 } } peers { permanent_uuid: "668165b874ca4fcb8714fa15f91de91b" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41173 } }
I20260926 00:59:45.970968 5696 consensus_queue.cc:260] T 55f09097eaad47b582fb9d5302849049 P 3bec1377543d433e832f77e9ebc6d740 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea28b33245c4862a7c4dfd2230860f7" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 40211 } } peers { permanent_uuid: "3bec1377543d433e832f77e9ebc6d740" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41097 } } peers { permanent_uuid: "668165b874ca4fcb8714fa15f91de91b" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41173 } }
I20260926 00:59:45.971045 5694 ts_tablet_manager.cc:1428] T 55f09097eaad47b582fb9d5302849049 P 668165b874ca4fcb8714fa15f91de91b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 00:59:45.970973 5695 raft_consensus.cc:348] T 55f09097eaad47b582fb9d5302849049 P 5ea28b33245c4862a7c4dfd2230860f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea28b33245c4862a7c4dfd2230860f7" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 40211 } } peers { permanent_uuid: "3bec1377543d433e832f77e9ebc6d740" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41097 } } peers { permanent_uuid: "668165b874ca4fcb8714fa15f91de91b" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41173 } }
I20260926 00:59:45.971170 5695 raft_consensus.cc:374] T 55f09097eaad47b582fb9d5302849049 P 5ea28b33245c4862a7c4dfd2230860f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 00:59:45.971204 5695 raft_consensus.cc:729] T 55f09097eaad47b582fb9d5302849049 P 5ea28b33245c4862a7c4dfd2230860f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ea28b33245c4862a7c4dfd2230860f7, State: Initialized, Role: FOLLOWER
I20260926 00:59:45.971257 5695 consensus_queue.cc:260] T 55f09097eaad47b582fb9d5302849049 P 5ea28b33245c4862a7c4dfd2230860f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea28b33245c4862a7c4dfd2230860f7" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 40211 } } peers { permanent_uuid: "3bec1377543d433e832f77e9ebc6d740" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41097 } } peers { permanent_uuid: "668165b874ca4fcb8714fa15f91de91b" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41173 } }
I20260926 00:59:45.971081 5696 ts_tablet_manager.cc:1428] T 55f09097eaad47b582fb9d5302849049 P 3bec1377543d433e832f77e9ebc6d740: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 00:59:45.971411 5695 ts_tablet_manager.cc:1428] T 55f09097eaad47b582fb9d5302849049 P 5ea28b33245c4862a7c4dfd2230860f7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 00:59:46.045814 5678 raft_consensus.cc:482] T 55f09097eaad47b582fb9d5302849049 P 5ea28b33245c4862a7c4dfd2230860f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 00:59:46.045917 5678 raft_consensus.cc:504] T 55f09097eaad47b582fb9d5302849049 P 5ea28b33245c4862a7c4dfd2230860f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea28b33245c4862a7c4dfd2230860f7" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 40211 } } peers { permanent_uuid: "3bec1377543d433e832f77e9ebc6d740" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41097 } } peers { permanent_uuid: "668165b874ca4fcb8714fa15f91de91b" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41173 } }
I20260926 00:59:46.046101 5678 leader_election.cc:288] T 55f09097eaad47b582fb9d5302849049 P 5ea28b33245c4862a7c4dfd2230860f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3bec1377543d433e832f77e9ebc6d740 (127.3.75.2:41097), 668165b874ca4fcb8714fa15f91de91b (127.3.75.3:41173)
I20260926 00:59:46.046283 5396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55f09097eaad47b582fb9d5302849049" candidate_uuid: "5ea28b33245c4862a7c4dfd2230860f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bec1377543d433e832f77e9ebc6d740" is_pre_election: true
I20260926 00:59:46.046296 5536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55f09097eaad47b582fb9d5302849049" candidate_uuid: "5ea28b33245c4862a7c4dfd2230860f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668165b874ca4fcb8714fa15f91de91b" is_pre_election: true
I20260926 00:59:46.046391 5396 raft_consensus.cc:2454] T 55f09097eaad47b582fb9d5302849049 P 3bec1377543d433e832f77e9ebc6d740 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ea28b33245c4862a7c4dfd2230860f7 in term 0.
I20260926 00:59:46.046391 5536 raft_consensus.cc:2454] T 55f09097eaad47b582fb9d5302849049 P 668165b874ca4fcb8714fa15f91de91b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ea28b33245c4862a7c4dfd2230860f7 in term 0.
I20260926 00:59:46.046573 5188 leader_election.cc:302] T 55f09097eaad47b582fb9d5302849049 P 5ea28b33245c4862a7c4dfd2230860f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ea28b33245c4862a7c4dfd2230860f7, 668165b874ca4fcb8714fa15f91de91b; no voters:
I20260926 00:59:46.046703 5678 raft_consensus.cc:2781] T 55f09097eaad47b582fb9d5302849049 P 5ea28b33245c4862a7c4dfd2230860f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 00:59:46.046762 5678 raft_consensus.cc:482] T 55f09097eaad47b582fb9d5302849049 P 5ea28b33245c4862a7c4dfd2230860f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 00:59:46.046777 5678 raft_consensus.cc:3037] T 55f09097eaad47b582fb9d5302849049 P 5ea28b33245c4862a7c4dfd2230860f7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 00:59:46.047528 5678 raft_consensus.cc:504] T 55f09097eaad47b582fb9d5302849049 P 5ea28b33245c4862a7c4dfd2230860f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea28b33245c4862a7c4dfd2230860f7" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 40211 } } peers { permanent_uuid: "3bec1377543d433e832f77e9ebc6d740" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41097 } } peers { permanent_uuid: "668165b874ca4fcb8714fa15f91de91b" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41173 } }
I20260926 00:59:46.047664 5678 leader_election.cc:288] T 55f09097eaad47b582fb9d5302849049 P 5ea28b33245c4862a7c4dfd2230860f7 [CANDIDATE]: Term 1 election: Requested vote from peers 3bec1377543d433e832f77e9ebc6d740 (127.3.75.2:41097), 668165b874ca4fcb8714fa15f91de91b (127.3.75.3:41173)
I20260926 00:59:46.047801 5396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55f09097eaad47b582fb9d5302849049" candidate_uuid: "5ea28b33245c4862a7c4dfd2230860f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bec1377543d433e832f77e9ebc6d740"
I20260926 00:59:46.047875 5396 raft_consensus.cc:3037] T 55f09097eaad47b582fb9d5302849049 P 3bec1377543d433e832f77e9ebc6d740 [term 0 FOLLOWER]: Advancing to term 1
I20260926 00:59:46.047811 5536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55f09097eaad47b582fb9d5302849049" candidate_uuid: "5ea28b33245c4862a7c4dfd2230860f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668165b874ca4fcb8714fa15f91de91b"
I20260926 00:59:46.047904 5536 raft_consensus.cc:3037] T 55f09097eaad47b582fb9d5302849049 P 668165b874ca4fcb8714fa15f91de91b [term 0 FOLLOWER]: Advancing to term 1
I20260926 00:59:46.048476 5536 raft_consensus.cc:2454] T 55f09097eaad47b582fb9d5302849049 P 668165b874ca4fcb8714fa15f91de91b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ea28b33245c4862a7c4dfd2230860f7 in term 1.
I20260926 00:59:46.048559 5396 raft_consensus.cc:2454] T 55f09097eaad47b582fb9d5302849049 P 3bec1377543d433e832f77e9ebc6d740 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ea28b33245c4862a7c4dfd2230860f7 in term 1.
I20260926 00:59:46.048663 5188 leader_election.cc:302] T 55f09097eaad47b582fb9d5302849049 P 5ea28b33245c4862a7c4dfd2230860f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ea28b33245c4862a7c4dfd2230860f7, 668165b874ca4fcb8714fa15f91de91b; no voters:
I20260926 00:59:46.048761 5678 raft_consensus.cc:2781] T 55f09097eaad47b582fb9d5302849049 P 5ea28b33245c4862a7c4dfd2230860f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 00:59:46.048825 5678 raft_consensus.cc:686] T 55f09097eaad47b582fb9d5302849049 P 5ea28b33245c4862a7c4dfd2230860f7 [term 1 LEADER]: Becoming Leader. State: Replica: 5ea28b33245c4862a7c4dfd2230860f7, State: Running, Role: LEADER
I20260926 00:59:46.048884 5678 consensus_queue.cc:237] T 55f09097eaad47b582fb9d5302849049 P 5ea28b33245c4862a7c4dfd2230860f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea28b33245c4862a7c4dfd2230860f7" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 40211 } } peers { permanent_uuid: "3bec1377543d433e832f77e9ebc6d740" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41097 } } peers { permanent_uuid: "668165b874ca4fcb8714fa15f91de91b" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41173 } }
I20260926 00:59:46.049559 5097 catalog_manager.cc:5495] T 55f09097eaad47b582fb9d5302849049 P 5ea28b33245c4862a7c4dfd2230860f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ea28b33245c4862a7c4dfd2230860f7 (127.3.75.1). New cstate: current_term: 1 leader_uuid: "5ea28b33245c4862a7c4dfd2230860f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea28b33245c4862a7c4dfd2230860f7" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 40211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3bec1377543d433e832f77e9ebc6d740" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668165b874ca4fcb8714fa15f91de91b" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41173 } health_report { overall_health: UNKNOWN } } }
I20260926 00:59:46.065675 5703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:46.063539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57008 (local address 127.3.75.1:40211)
0926 00:59:46.063702 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:46.063704 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:46.063713 (+ 9us) server_negotiation.cc:407] Connection header received
0926 00:59:46.063736 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:46.063737 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:46.063749 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:46.063820 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 00:59:46.063922 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:46.064390 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:46.065096 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:46.065290 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:46.065340 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:46.065446 (+ 106us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 00:59:46.065469 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 00:59:46.065504 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:46.065552 (+ 48us) server_negotiation.cc:299] Negotiation successful
0926 00:59:46.065567 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":93,"threads_started":1}
I20260926 00:59:46.067782 5536 raft_consensus.cc:1261] T 55f09097eaad47b582fb9d5302849049 P 668165b874ca4fcb8714fa15f91de91b [term 1 FOLLOWER]: Refusing update from remote peer 5ea28b33245c4862a7c4dfd2230860f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 00:59:46.067830 5396 raft_consensus.cc:1261] T 55f09097eaad47b582fb9d5302849049 P 3bec1377543d433e832f77e9ebc6d740 [term 1 FOLLOWER]: Refusing update from remote peer 5ea28b33245c4862a7c4dfd2230860f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 00:59:46.068049 5674 consensus_queue.cc:1035] T 55f09097eaad47b582fb9d5302849049 P 5ea28b33245c4862a7c4dfd2230860f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668165b874ca4fcb8714fa15f91de91b" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 00:59:46.068136 5678 consensus_queue.cc:1035] T 55f09097eaad47b582fb9d5302849049 P 5ea28b33245c4862a7c4dfd2230860f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bec1377543d433e832f77e9ebc6d740" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 00:59:46.077379 5098 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 00:59:46.077461 5098 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 00:59:46.078055 5098 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b5d54f69067e4c24a51d89252d078c71: Sending DeleteTablet for 3 replicas of tablet 55f09097eaad47b582fb9d5302849049
I20260926 00:59:46.078310 5516 tablet_service.cc:1514] Processing DeleteTablet for tablet 55f09097eaad47b582fb9d5302849049 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 00:59:46 UTC) from {username='kudu', principal='kudu/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:54370
I20260926 00:59:46.078346 5375 tablet_service.cc:1514] Processing DeleteTablet for tablet 55f09097eaad47b582fb9d5302849049 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 00:59:46 UTC) from {username='kudu', principal='kudu/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:40564
I20260926 00:59:46.078413 5236 tablet_service.cc:1514] Processing DeleteTablet for tablet 55f09097eaad47b582fb9d5302849049 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 00:59:46 UTC) from {username='kudu', principal='kudu/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:56984
I20260926 00:59:46.078440 5684 tablet_replica.cc:330] T 55f09097eaad47b582fb9d5302849049 P 3bec1377543d433e832f77e9ebc6d740: stopping tablet replica
I20260926 00:59:46.078493 5684 raft_consensus.cc:2229] T 55f09097eaad47b582fb9d5302849049 P 3bec1377543d433e832f77e9ebc6d740 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 00:59:46.078500 5683 tablet_replica.cc:330] T 55f09097eaad47b582fb9d5302849049 P 668165b874ca4fcb8714fa15f91de91b: stopping tablet replica
I20260926 00:59:46.078539 5684 raft_consensus.cc:2258] T 55f09097eaad47b582fb9d5302849049 P 3bec1377543d433e832f77e9ebc6d740 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 00:59:46.078548 5683 raft_consensus.cc:2229] T 55f09097eaad47b582fb9d5302849049 P 668165b874ca4fcb8714fa15f91de91b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 00:59:46.078500 5098 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58236:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 00:59:46.078583 5683 raft_consensus.cc:2258] T 55f09097eaad47b582fb9d5302849049 P 668165b874ca4fcb8714fa15f91de91b [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 00:59:46.078634 5098 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 00:59:46.078788 5685 tablet_replica.cc:330] T 55f09097eaad47b582fb9d5302849049 P 5ea28b33245c4862a7c4dfd2230860f7: stopping tablet replica
I20260926 00:59:46.078842 5685 raft_consensus.cc:2229] T 55f09097eaad47b582fb9d5302849049 P 5ea28b33245c4862a7c4dfd2230860f7 [term 1 LEADER]: Raft consensus shutting down.
I20260926 00:59:46.078862 5684 ts_tablet_manager.cc:1905] T 55f09097eaad47b582fb9d5302849049 P 3bec1377543d433e832f77e9ebc6d740: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 00:59:46.078912 5685 raft_consensus.cc:2258] T 55f09097eaad47b582fb9d5302849049 P 5ea28b33245c4862a7c4dfd2230860f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 00:59:46.079008 5683 ts_tablet_manager.cc:1905] T 55f09097eaad47b582fb9d5302849049 P 668165b874ca4fcb8714fa15f91de91b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 00:59:46.079182 5685 ts_tablet_manager.cc:1905] T 55f09097eaad47b582fb9d5302849049 P 5ea28b33245c4862a7c4dfd2230860f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 00:59:46.080559 5684 ts_tablet_manager.cc:1918] T 55f09097eaad47b582fb9d5302849049 P 3bec1377543d433e832f77e9ebc6d740: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 00:59:46.080614 5684 log.cc:1196] T 55f09097eaad47b582fb9d5302849049 P 3bec1377543d433e832f77e9ebc6d740: Deleting WAL directory at /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/ts-1/wal/wals/55f09097eaad47b582fb9d5302849049
I20260926 00:59:46.080677 5685 ts_tablet_manager.cc:1918] T 55f09097eaad47b582fb9d5302849049 P 5ea28b33245c4862a7c4dfd2230860f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 00:59:46.080732 5685 log.cc:1196] T 55f09097eaad47b582fb9d5302849049 P 5ea28b33245c4862a7c4dfd2230860f7: Deleting WAL directory at /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/ts-0/wal/wals/55f09097eaad47b582fb9d5302849049
I20260926 00:59:46.080837 5684 ts_tablet_manager.cc:1939] T 55f09097eaad47b582fb9d5302849049 P 3bec1377543d433e832f77e9ebc6d740: Deleting consensus metadata
I20260926 00:59:46.080971 5685 ts_tablet_manager.cc:1939] T 55f09097eaad47b582fb9d5302849049 P 5ea28b33245c4862a7c4dfd2230860f7: Deleting consensus metadata
I20260926 00:59:46.081178 5683 ts_tablet_manager.cc:1918] T 55f09097eaad47b582fb9d5302849049 P 668165b874ca4fcb8714fa15f91de91b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 00:59:46.081231 5683 log.cc:1196] T 55f09097eaad47b582fb9d5302849049 P 668165b874ca4fcb8714fa15f91de91b: Deleting WAL directory at /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/ts-2/wal/wals/55f09097eaad47b582fb9d5302849049
I20260926 00:59:46.081292 5085 catalog_manager.cc:4841] TS 5ea28b33245c4862a7c4dfd2230860f7 (127.3.75.1:40211): tablet 55f09097eaad47b582fb9d5302849049 (table test-table [id=7eda5a747b13430e8d797abf70614484]) successfully deleted
I20260926 00:59:46.081359 5236 tablet_service.cc:1467] Processing CreateTablet for tablet b84b661ad6874f95bbb41c998ea9d7c3 (DEFAULT_TABLE table=test-table [id=9afe5d2a316444fd8a2a403b5173b0b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 00:59:46.081444 5683 ts_tablet_manager.cc:1939] T 55f09097eaad47b582fb9d5302849049 P 668165b874ca4fcb8714fa15f91de91b: Deleting consensus metadata
I20260926 00:59:46.081460 5236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b84b661ad6874f95bbb41c998ea9d7c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 00:59:46.081488 5516 tablet_service.cc:1467] Processing CreateTablet for tablet b84b661ad6874f95bbb41c998ea9d7c3 (DEFAULT_TABLE table=test-table [id=9afe5d2a316444fd8a2a403b5173b0b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 00:59:46.081574 5516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b84b661ad6874f95bbb41c998ea9d7c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 00:59:46.082662 5695 tablet_bootstrap.cc:492] T b84b661ad6874f95bbb41c998ea9d7c3 P 5ea28b33245c4862a7c4dfd2230860f7: Bootstrap starting.
I20260926 00:59:46.082742 5694 tablet_bootstrap.cc:492] T b84b661ad6874f95bbb41c998ea9d7c3 P 668165b874ca4fcb8714fa15f91de91b: Bootstrap starting.
I20260926 00:59:46.082906 5375 tablet_service.cc:1467] Processing CreateTablet for tablet b84b661ad6874f95bbb41c998ea9d7c3 (DEFAULT_TABLE table=test-table [id=9afe5d2a316444fd8a2a403b5173b0b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 00:59:46.083011 5375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b84b661ad6874f95bbb41c998ea9d7c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 00:59:46.083065 5695 tablet_bootstrap.cc:654] T b84b661ad6874f95bbb41c998ea9d7c3 P 5ea28b33245c4862a7c4dfd2230860f7: Neither blocks nor log segments found. Creating new log.
I20260926 00:59:46.083113 5694 tablet_bootstrap.cc:654] T b84b661ad6874f95bbb41c998ea9d7c3 P 668165b874ca4fcb8714fa15f91de91b: Neither blocks nor log segments found. Creating new log.
I20260926 00:59:46.083333 5084 catalog_manager.cc:4841] TS 668165b874ca4fcb8714fa15f91de91b (127.3.75.3:41173): tablet 55f09097eaad47b582fb9d5302849049 (table test-table [id=7eda5a747b13430e8d797abf70614484]) successfully deleted
I20260926 00:59:46.083683 5695 tablet_bootstrap.cc:492] T b84b661ad6874f95bbb41c998ea9d7c3 P 5ea28b33245c4862a7c4dfd2230860f7: No bootstrap required, opened a new log
I20260926 00:59:46.083750 5695 ts_tablet_manager.cc:1397] T b84b661ad6874f95bbb41c998ea9d7c3 P 5ea28b33245c4862a7c4dfd2230860f7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 00:59:46.083859 5695 raft_consensus.cc:348] T b84b661ad6874f95bbb41c998ea9d7c3 P 5ea28b33245c4862a7c4dfd2230860f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea28b33245c4862a7c4dfd2230860f7" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 40211 } } peers { permanent_uuid: "3bec1377543d433e832f77e9ebc6d740" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41097 } } peers { permanent_uuid: "668165b874ca4fcb8714fa15f91de91b" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41173 } }
I20260926 00:59:46.083938 5695 raft_consensus.cc:374] T b84b661ad6874f95bbb41c998ea9d7c3 P 5ea28b33245c4862a7c4dfd2230860f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 00:59:46.083973 5695 raft_consensus.cc:729] T b84b661ad6874f95bbb41c998ea9d7c3 P 5ea28b33245c4862a7c4dfd2230860f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ea28b33245c4862a7c4dfd2230860f7, State: Initialized, Role: FOLLOWER
I20260926 00:59:46.084020 5695 consensus_queue.cc:260] T b84b661ad6874f95bbb41c998ea9d7c3 P 5ea28b33245c4862a7c4dfd2230860f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea28b33245c4862a7c4dfd2230860f7" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 40211 } } peers { permanent_uuid: "3bec1377543d433e832f77e9ebc6d740" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41097 } } peers { permanent_uuid: "668165b874ca4fcb8714fa15f91de91b" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41173 } }
I20260926 00:59:46.084111 5695 ts_tablet_manager.cc:1428] T b84b661ad6874f95bbb41c998ea9d7c3 P 5ea28b33245c4862a7c4dfd2230860f7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:46.084141 5696 tablet_bootstrap.cc:492] T b84b661ad6874f95bbb41c998ea9d7c3 P 3bec1377543d433e832f77e9ebc6d740: Bootstrap starting.
I20260926 00:59:46.084136 5085 catalog_manager.cc:4841] TS 3bec1377543d433e832f77e9ebc6d740 (127.3.75.2:41097): tablet 55f09097eaad47b582fb9d5302849049 (table test-table [id=7eda5a747b13430e8d797abf70614484]) successfully deleted
I20260926 00:59:46.084447 5696 tablet_bootstrap.cc:654] T b84b661ad6874f95bbb41c998ea9d7c3 P 3bec1377543d433e832f77e9ebc6d740: Neither blocks nor log segments found. Creating new log.
I20260926 00:59:46.084601 5694 tablet_bootstrap.cc:492] T b84b661ad6874f95bbb41c998ea9d7c3 P 668165b874ca4fcb8714fa15f91de91b: No bootstrap required, opened a new log
I20260926 00:59:46.084647 5694 ts_tablet_manager.cc:1397] T b84b661ad6874f95bbb41c998ea9d7c3 P 668165b874ca4fcb8714fa15f91de91b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 00:59:46.084792 5694 raft_consensus.cc:348] T b84b661ad6874f95bbb41c998ea9d7c3 P 668165b874ca4fcb8714fa15f91de91b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea28b33245c4862a7c4dfd2230860f7" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 40211 } } peers { permanent_uuid: "3bec1377543d433e832f77e9ebc6d740" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41097 } } peers { permanent_uuid: "668165b874ca4fcb8714fa15f91de91b" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41173 } }
I20260926 00:59:46.084867 5694 raft_consensus.cc:374] T b84b661ad6874f95bbb41c998ea9d7c3 P 668165b874ca4fcb8714fa15f91de91b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 00:59:46.084892 5694 raft_consensus.cc:729] T b84b661ad6874f95bbb41c998ea9d7c3 P 668165b874ca4fcb8714fa15f91de91b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668165b874ca4fcb8714fa15f91de91b, State: Initialized, Role: FOLLOWER
I20260926 00:59:46.084959 5696 tablet_bootstrap.cc:492] T b84b661ad6874f95bbb41c998ea9d7c3 P 3bec1377543d433e832f77e9ebc6d740: No bootstrap required, opened a new log
I20260926 00:59:46.084933 5694 consensus_queue.cc:260] T b84b661ad6874f95bbb41c998ea9d7c3 P 668165b874ca4fcb8714fa15f91de91b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea28b33245c4862a7c4dfd2230860f7" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 40211 } } peers { permanent_uuid: "3bec1377543d433e832f77e9ebc6d740" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41097 } } peers { permanent_uuid: "668165b874ca4fcb8714fa15f91de91b" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41173 } }
I20260926 00:59:46.085009 5696 ts_tablet_manager.cc:1397] T b84b661ad6874f95bbb41c998ea9d7c3 P 3bec1377543d433e832f77e9ebc6d740: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 00:59:46.085040 5694 ts_tablet_manager.cc:1428] T b84b661ad6874f95bbb41c998ea9d7c3 P 668165b874ca4fcb8714fa15f91de91b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 00:59:46.085119 5696 raft_consensus.cc:348] T b84b661ad6874f95bbb41c998ea9d7c3 P 3bec1377543d433e832f77e9ebc6d740 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea28b33245c4862a7c4dfd2230860f7" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 40211 } } peers { permanent_uuid: "3bec1377543d433e832f77e9ebc6d740" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41097 } } peers { permanent_uuid: "668165b874ca4fcb8714fa15f91de91b" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41173 } }
I20260926 00:59:46.085206 5696 raft_consensus.cc:374] T b84b661ad6874f95bbb41c998ea9d7c3 P 3bec1377543d433e832f77e9ebc6d740 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 00:59:46.085232 5696 raft_consensus.cc:729] T b84b661ad6874f95bbb41c998ea9d7c3 P 3bec1377543d433e832f77e9ebc6d740 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bec1377543d433e832f77e9ebc6d740, State: Initialized, Role: FOLLOWER
I20260926 00:59:46.085296 5696 consensus_queue.cc:260] T b84b661ad6874f95bbb41c998ea9d7c3 P 3bec1377543d433e832f77e9ebc6d740 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea28b33245c4862a7c4dfd2230860f7" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 40211 } } peers { permanent_uuid: "3bec1377543d433e832f77e9ebc6d740" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41097 } } peers { permanent_uuid: "668165b874ca4fcb8714fa15f91de91b" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41173 } }
I20260926 00:59:46.085387 5696 ts_tablet_manager.cc:1428] T b84b661ad6874f95bbb41c998ea9d7c3 P 3bec1377543d433e832f77e9ebc6d740: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:46.309007 5641 raft_consensus.cc:482] T b84b661ad6874f95bbb41c998ea9d7c3 P 3bec1377543d433e832f77e9ebc6d740 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 00:59:46.309099 5641 raft_consensus.cc:504] T b84b661ad6874f95bbb41c998ea9d7c3 P 3bec1377543d433e832f77e9ebc6d740 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea28b33245c4862a7c4dfd2230860f7" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 40211 } } peers { permanent_uuid: "3bec1377543d433e832f77e9ebc6d740" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41097 } } peers { permanent_uuid: "668165b874ca4fcb8714fa15f91de91b" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41173 } }
I20260926 00:59:46.309303 5641 leader_election.cc:288] T b84b661ad6874f95bbb41c998ea9d7c3 P 3bec1377543d433e832f77e9ebc6d740 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ea28b33245c4862a7c4dfd2230860f7 (127.3.75.1:40211), 668165b874ca4fcb8714fa15f91de91b (127.3.75.3:41173)
I20260926 00:59:46.309494 5256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84b661ad6874f95bbb41c998ea9d7c3" candidate_uuid: "3bec1377543d433e832f77e9ebc6d740" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ea28b33245c4862a7c4dfd2230860f7" is_pre_election: true
I20260926 00:59:46.309512 5536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84b661ad6874f95bbb41c998ea9d7c3" candidate_uuid: "3bec1377543d433e832f77e9ebc6d740" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668165b874ca4fcb8714fa15f91de91b" is_pre_election: true
I20260926 00:59:46.309594 5256 raft_consensus.cc:2454] T b84b661ad6874f95bbb41c998ea9d7c3 P 5ea28b33245c4862a7c4dfd2230860f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bec1377543d433e832f77e9ebc6d740 in term 0.
I20260926 00:59:46.309624 5536 raft_consensus.cc:2454] T b84b661ad6874f95bbb41c998ea9d7c3 P 668165b874ca4fcb8714fa15f91de91b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bec1377543d433e832f77e9ebc6d740 in term 0.
I20260926 00:59:46.309755 5329 leader_election.cc:302] T b84b661ad6874f95bbb41c998ea9d7c3 P 3bec1377543d433e832f77e9ebc6d740 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bec1377543d433e832f77e9ebc6d740, 5ea28b33245c4862a7c4dfd2230860f7; no voters:
I20260926 00:59:46.309860 5641 raft_consensus.cc:2781] T b84b661ad6874f95bbb41c998ea9d7c3 P 3bec1377543d433e832f77e9ebc6d740 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 00:59:46.309911 5641 raft_consensus.cc:482] T b84b661ad6874f95bbb41c998ea9d7c3 P 3bec1377543d433e832f77e9ebc6d740 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 00:59:46.309926 5641 raft_consensus.cc:3037] T b84b661ad6874f95bbb41c998ea9d7c3 P 3bec1377543d433e832f77e9ebc6d740 [term 0 FOLLOWER]: Advancing to term 1
I20260926 00:59:46.310580 5641 raft_consensus.cc:504] T b84b661ad6874f95bbb41c998ea9d7c3 P 3bec1377543d433e832f77e9ebc6d740 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea28b33245c4862a7c4dfd2230860f7" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 40211 } } peers { permanent_uuid: "3bec1377543d433e832f77e9ebc6d740" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41097 } } peers { permanent_uuid: "668165b874ca4fcb8714fa15f91de91b" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41173 } }
I20260926 00:59:46.310717 5641 leader_election.cc:288] T b84b661ad6874f95bbb41c998ea9d7c3 P 3bec1377543d433e832f77e9ebc6d740 [CANDIDATE]: Term 1 election: Requested vote from peers 5ea28b33245c4862a7c4dfd2230860f7 (127.3.75.1:40211), 668165b874ca4fcb8714fa15f91de91b (127.3.75.3:41173)
I20260926 00:59:46.310849 5536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84b661ad6874f95bbb41c998ea9d7c3" candidate_uuid: "3bec1377543d433e832f77e9ebc6d740" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668165b874ca4fcb8714fa15f91de91b"
I20260926 00:59:46.310910 5256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84b661ad6874f95bbb41c998ea9d7c3" candidate_uuid: "3bec1377543d433e832f77e9ebc6d740" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ea28b33245c4862a7c4dfd2230860f7"
I20260926 00:59:46.310966 5536 raft_consensus.cc:3037] T b84b661ad6874f95bbb41c998ea9d7c3 P 668165b874ca4fcb8714fa15f91de91b [term 0 FOLLOWER]: Advancing to term 1
I20260926 00:59:46.310976 5256 raft_consensus.cc:3037] T b84b661ad6874f95bbb41c998ea9d7c3 P 5ea28b33245c4862a7c4dfd2230860f7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 00:59:46.311705 5536 raft_consensus.cc:2454] T b84b661ad6874f95bbb41c998ea9d7c3 P 668165b874ca4fcb8714fa15f91de91b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bec1377543d433e832f77e9ebc6d740 in term 1.
I20260926 00:59:46.311705 5256 raft_consensus.cc:2454] T b84b661ad6874f95bbb41c998ea9d7c3 P 5ea28b33245c4862a7c4dfd2230860f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bec1377543d433e832f77e9ebc6d740 in term 1.
I20260926 00:59:46.311880 5329 leader_election.cc:302] T b84b661ad6874f95bbb41c998ea9d7c3 P 3bec1377543d433e832f77e9ebc6d740 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bec1377543d433e832f77e9ebc6d740, 5ea28b33245c4862a7c4dfd2230860f7; no voters:
I20260926 00:59:46.311990 5641 raft_consensus.cc:2781] T b84b661ad6874f95bbb41c998ea9d7c3 P 3bec1377543d433e832f77e9ebc6d740 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 00:59:46.312043 5641 raft_consensus.cc:686] T b84b661ad6874f95bbb41c998ea9d7c3 P 3bec1377543d433e832f77e9ebc6d740 [term 1 LEADER]: Becoming Leader. State: Replica: 3bec1377543d433e832f77e9ebc6d740, State: Running, Role: LEADER
I20260926 00:59:46.312095 5641 consensus_queue.cc:237] T b84b661ad6874f95bbb41c998ea9d7c3 P 3bec1377543d433e832f77e9ebc6d740 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea28b33245c4862a7c4dfd2230860f7" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 40211 } } peers { permanent_uuid: "3bec1377543d433e832f77e9ebc6d740" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41097 } } peers { permanent_uuid: "668165b874ca4fcb8714fa15f91de91b" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41173 } }
I20260926 00:59:46.312640 5098 catalog_manager.cc:5495] T b84b661ad6874f95bbb41c998ea9d7c3 P 3bec1377543d433e832f77e9ebc6d740 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3bec1377543d433e832f77e9ebc6d740 (127.3.75.2). New cstate: current_term: 1 leader_uuid: "3bec1377543d433e832f77e9ebc6d740" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea28b33245c4862a7c4dfd2230860f7" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 40211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3bec1377543d433e832f77e9ebc6d740" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668165b874ca4fcb8714fa15f91de91b" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41173 } health_report { overall_health: UNKNOWN } } }
I20260926 00:59:46.369985 5711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:46.367918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40598 (local address 127.3.75.2:41097)
0926 00:59:46.368062 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:46.368064 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:46.368071 (+ 7us) server_negotiation.cc:407] Connection header received
0926 00:59:46.368089 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:46.368089 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:46.368098 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:46.368154 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 00:59:46.368265 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:46.368740 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:46.369390 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:46.369580 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:46.369634 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:46.369741 (+ 107us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 00:59:46.369764 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 00:59:46.369809 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:46.369862 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 00:59:46.369879 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":83,"threads_started":1}
I20260926 00:59:46.372269 5536 raft_consensus.cc:1261] T b84b661ad6874f95bbb41c998ea9d7c3 P 668165b874ca4fcb8714fa15f91de91b [term 1 FOLLOWER]: Refusing update from remote peer 3bec1377543d433e832f77e9ebc6d740: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 00:59:46.372293 5256 raft_consensus.cc:1261] T b84b661ad6874f95bbb41c998ea9d7c3 P 5ea28b33245c4862a7c4dfd2230860f7 [term 1 FOLLOWER]: Refusing update from remote peer 3bec1377543d433e832f77e9ebc6d740: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 00:59:46.372537 5641 consensus_queue.cc:1035] T b84b661ad6874f95bbb41c998ea9d7c3 P 3bec1377543d433e832f77e9ebc6d740 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668165b874ca4fcb8714fa15f91de91b" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 00:59:46.372658 5664 consensus_queue.cc:1035] T b84b661ad6874f95bbb41c998ea9d7c3 P 3bec1377543d433e832f77e9ebc6d740 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ea28b33245c4862a7c4dfd2230860f7" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 40211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 00:59:46.376765 5662 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 00:59:47.893018 5098 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 00:59:47.893154 5098 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 00:59:47.893906 5098 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b5d54f69067e4c24a51d89252d078c71: Sending DeleteTablet for 3 replicas of tablet b84b661ad6874f95bbb41c998ea9d7c3
I20260926 00:59:47.894174 5516 tablet_service.cc:1514] Processing DeleteTablet for tablet b84b661ad6874f95bbb41c998ea9d7c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 00:59:47 UTC) from {username='kudu', principal='kudu/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:54370
I20260926 00:59:47.894217 5236 tablet_service.cc:1514] Processing DeleteTablet for tablet b84b661ad6874f95bbb41c998ea9d7c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 00:59:47 UTC) from {username='kudu', principal='kudu/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:56984
I20260926 00:59:47.894354 5375 tablet_service.cc:1514] Processing DeleteTablet for tablet b84b661ad6874f95bbb41c998ea9d7c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 00:59:47 UTC) from {username='kudu', principal='kudu/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:40564
I20260926 00:59:47.894385 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 5171
I20260926 00:59:47.894388 5735 tablet_replica.cc:330] T b84b661ad6874f95bbb41c998ea9d7c3 P 668165b874ca4fcb8714fa15f91de91b: stopping tablet replica
I20260926 00:59:47.895151 5735 raft_consensus.cc:2229] T b84b661ad6874f95bbb41c998ea9d7c3 P 668165b874ca4fcb8714fa15f91de91b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 00:59:47.895226 5735 raft_consensus.cc:2258] T b84b661ad6874f95bbb41c998ea9d7c3 P 668165b874ca4fcb8714fa15f91de91b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 00:59:47.895464 5737 tablet_replica.cc:330] T b84b661ad6874f95bbb41c998ea9d7c3 P 3bec1377543d433e832f77e9ebc6d740: stopping tablet replica
I20260926 00:59:47.895510 5737 raft_consensus.cc:2229] T b84b661ad6874f95bbb41c998ea9d7c3 P 3bec1377543d433e832f77e9ebc6d740 [term 1 LEADER]: Raft consensus shutting down.
I20260926 00:59:47.895502 5735 ts_tablet_manager.cc:1905] T b84b661ad6874f95bbb41c998ea9d7c3 P 668165b874ca4fcb8714fa15f91de91b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 00:59:47.895550 5737 raft_consensus.cc:2258] T b84b661ad6874f95bbb41c998ea9d7c3 P 3bec1377543d433e832f77e9ebc6d740 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 00:59:47.895742 5737 ts_tablet_manager.cc:1905] T b84b661ad6874f95bbb41c998ea9d7c3 P 3bec1377543d433e832f77e9ebc6d740: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 00:59:47.897352 5737 ts_tablet_manager.cc:1918] T b84b661ad6874f95bbb41c998ea9d7c3 P 3bec1377543d433e832f77e9ebc6d740: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 00:59:47.897393 5737 log.cc:1196] T b84b661ad6874f95bbb41c998ea9d7c3 P 3bec1377543d433e832f77e9ebc6d740: Deleting WAL directory at /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/ts-1/wal/wals/b84b661ad6874f95bbb41c998ea9d7c3
I20260926 00:59:47.897573 5737 ts_tablet_manager.cc:1939] T b84b661ad6874f95bbb41c998ea9d7c3 P 3bec1377543d433e832f77e9ebc6d740: Deleting consensus metadata
I20260926 00:59:47.897576 5735 ts_tablet_manager.cc:1918] T b84b661ad6874f95bbb41c998ea9d7c3 P 668165b874ca4fcb8714fa15f91de91b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 00:59:47.897632 5735 log.cc:1196] T b84b661ad6874f95bbb41c998ea9d7c3 P 668165b874ca4fcb8714fa15f91de91b: Deleting WAL directory at /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384372073861-3372-0/minicluster-data/ts-2/wal/wals/b84b661ad6874f95bbb41c998ea9d7c3
I20260926 00:59:47.897854 5735 ts_tablet_manager.cc:1939] T b84b661ad6874f95bbb41c998ea9d7c3 P 668165b874ca4fcb8714fa15f91de91b: Deleting consensus metadata
I20260926 00:59:47.897845 5085 catalog_manager.cc:4841] TS 3bec1377543d433e832f77e9ebc6d740 (127.3.75.2:41097): tablet b84b661ad6874f95bbb41c998ea9d7c3 (table test-table [id=9afe5d2a316444fd8a2a403b5173b0b8]) successfully deleted
I20260926 00:59:47.898313 5084 catalog_manager.cc:4841] TS 668165b874ca4fcb8714fa15f91de91b (127.3.75.3:41173): tablet b84b661ad6874f95bbb41c998ea9d7c3 (table test-table [id=9afe5d2a316444fd8a2a403b5173b0b8]) successfully deleted
W20260926 00:59:47.902413 5085 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.75.1:40211 (error 108)
I20260926 00:59:47.903034 5739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:47.902814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.1:40211 (local address 127.0.0.1:57024)
0926 00:59:47.902946 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:47.903003 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.1:40211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":91,"threads_started":1}
I20260926 00:59:47.903023 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 5311
W20260926 00:59:47.904547 5085 catalog_manager.cc:4584] TS 5ea28b33245c4862a7c4dfd2230860f7 (127.3.75.1:40211): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b84b661ad6874f95bbb41c998ea9d7c3: Network error: Client connection negotiation failed: client connection to 127.3.75.1:40211: connect: Connection refused (error 111)
I20260926 00:59:47.909267 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 5451
I20260926 00:59:47.914008 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 5067
2026-09-26T00:59:47Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5658 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 00:59:47 dist-test-slave-7v10 krb5kdc[5742](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 00:59:47 dist-test-slave-7v10 krb5kdc[5742](info): set up 2 sockets
Sep 26 00:59:47 dist-test-slave-7v10 krb5kdc[5742](info): commencing operation
krb5kdc: starting...
W20260926 00:59:48.941672 3372 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Sep 26 00:59:49 dist-test-slave-7v10 krb5kdc[5742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T00:59:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T00:59:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.75.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab.
Entry for principal kudu/127.3.75.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab.
I20260926 00:59:49.055166 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:44033
--webserver_interface=127.3.75.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:37065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.75.62:44033
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 00:59:49.134099 5755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 00:59:49.134315 5755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 00:59:49.134351 5755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 00:59:49.135542 5755 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 00:59:49.135605 5755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 00:59:49.135628 5755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 00:59:49.135655 5755 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 00:59:49.135685 5755 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 00:59:49.137071 5755 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:37065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.75.62:44033
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:44033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 00:59:49.137323 5755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 00:59:49.137542 5755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 00:59:49.142192 5760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:49.142220 5763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:49.142222 5761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 00:59:49.142421 5755 server_base.cc:1034] running on GCE node
I20260926 00:59:49.142652 5755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 00:59:49.142974 5755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 00:59:49.144132 5755 hybrid_clock.cc:648] HybridClock initialized: now 1790384389144128 us; error 34 us; skew 500 ppm
Sep 26 00:59:49 dist-test-slave-7v10 krb5kdc[5742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384389, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 00:59:49.146440 5755 init.cc:377] Logged in from keytab as kudu/127.3.75.62@KRBTEST.COM (short username kudu)
I20260926 00:59:49.146804 5755 webserver.cc:460] Webserver started at http://127.3.75.62:44731/ using document root <none> and password file <none>
I20260926 00:59:49.147063 5755 fs_manager.cc:359] Metadata directory not provided
I20260926 00:59:49.147109 5755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 00:59:49.147197 5755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 00:59:49.148152 5755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/master-0/data/instance:
uuid: "eae818882c4a4f439ca4948ee1ad0d3d"
format_stamp: "Formatted at 2026-09-26 00:59:49 on dist-test-slave-7v10"
server_key: "be724a021dc6d661ee0e5c9808add3ad"
server_key_iv: "086acf7e8d4cd3d1069189456221427f"
server_key_version: "encryptionkey@0"
I20260926 00:59:49.148437 5755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/master-0/wal/instance:
uuid: "eae818882c4a4f439ca4948ee1ad0d3d"
format_stamp: "Formatted at 2026-09-26 00:59:49 on dist-test-slave-7v10"
server_key: "be724a021dc6d661ee0e5c9808add3ad"
server_key_iv: "086acf7e8d4cd3d1069189456221427f"
server_key_version: "encryptionkey@0"
I20260926 00:59:49.149653 5755 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 00:59:49.150321 5770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:49.150512 5755 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 00:59:49.150600 5755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/master-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/master-0/wal
uuid: "eae818882c4a4f439ca4948ee1ad0d3d"
format_stamp: "Formatted at 2026-09-26 00:59:49 on dist-test-slave-7v10"
server_key: "be724a021dc6d661ee0e5c9808add3ad"
server_key_iv: "086acf7e8d4cd3d1069189456221427f"
server_key_version: "encryptionkey@0"
I20260926 00:59:49.150651 5755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 00:59:49.163964 5755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 00:59:49.165633 5755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 00:59:49.165735 5755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 00:59:49.169378 5755 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.62:44033
I20260926 00:59:49.169473 5822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.62:44033 every 8 connection(s)
I20260926 00:59:49.169761 5755 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
I20260926 00:59:49.170600 5823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 00:59:49.173012 5823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eae818882c4a4f439ca4948ee1ad0d3d: Bootstrap starting.
I20260926 00:59:49.173575 5823 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eae818882c4a4f439ca4948ee1ad0d3d: Neither blocks nor log segments found. Creating new log.
I20260926 00:59:49.173871 5823 log.cc:824] T 00000000000000000000000000000000 P eae818882c4a4f439ca4948ee1ad0d3d: Log is configured to *not* fsync() on all Append() calls
I20260926 00:59:49.174505 5823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eae818882c4a4f439ca4948ee1ad0d3d: No bootstrap required, opened a new log
I20260926 00:59:49.176059 5823 raft_consensus.cc:348] T 00000000000000000000000000000000 P eae818882c4a4f439ca4948ee1ad0d3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eae818882c4a4f439ca4948ee1ad0d3d" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 44033 } }
I20260926 00:59:49.176182 5823 raft_consensus.cc:374] T 00000000000000000000000000000000 P eae818882c4a4f439ca4948ee1ad0d3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 00:59:49.176203 5823 raft_consensus.cc:729] T 00000000000000000000000000000000 P eae818882c4a4f439ca4948ee1ad0d3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eae818882c4a4f439ca4948ee1ad0d3d, State: Initialized, Role: FOLLOWER
I20260926 00:59:49.176376 5823 consensus_queue.cc:260] T 00000000000000000000000000000000 P eae818882c4a4f439ca4948ee1ad0d3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eae818882c4a4f439ca4948ee1ad0d3d" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 44033 } }
I20260926 00:59:49.176455 5823 raft_consensus.cc:388] T 00000000000000000000000000000000 P eae818882c4a4f439ca4948ee1ad0d3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 00:59:49.176508 5823 raft_consensus.cc:482] T 00000000000000000000000000000000 P eae818882c4a4f439ca4948ee1ad0d3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 00:59:49.176556 5823 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eae818882c4a4f439ca4948ee1ad0d3d [term 0 FOLLOWER]: Advancing to term 1
I20260926 00:59:49.177026 5823 raft_consensus.cc:504] T 00000000000000000000000000000000 P eae818882c4a4f439ca4948ee1ad0d3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eae818882c4a4f439ca4948ee1ad0d3d" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 44033 } }
I20260926 00:59:49.177125 5823 leader_election.cc:302] T 00000000000000000000000000000000 P eae818882c4a4f439ca4948ee1ad0d3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eae818882c4a4f439ca4948ee1ad0d3d; no voters:
I20260926 00:59:49.177295 5823 leader_election.cc:288] T 00000000000000000000000000000000 P eae818882c4a4f439ca4948ee1ad0d3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 00:59:49.177333 5826 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eae818882c4a4f439ca4948ee1ad0d3d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 00:59:49.177542 5823 sys_catalog.cc:564] T 00000000000000000000000000000000 P eae818882c4a4f439ca4948ee1ad0d3d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 00:59:49.177676 5826 raft_consensus.cc:686] T 00000000000000000000000000000000 P eae818882c4a4f439ca4948ee1ad0d3d [term 1 LEADER]: Becoming Leader. State: Replica: eae818882c4a4f439ca4948ee1ad0d3d, State: Running, Role: LEADER
I20260926 00:59:49.177825 5826 consensus_queue.cc:237] T 00000000000000000000000000000000 P eae818882c4a4f439ca4948ee1ad0d3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eae818882c4a4f439ca4948ee1ad0d3d" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 44033 } }
I20260926 00:59:49.178190 5828 sys_catalog.cc:455] T 00000000000000000000000000000000 P eae818882c4a4f439ca4948ee1ad0d3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader eae818882c4a4f439ca4948ee1ad0d3d. Latest consensus state: current_term: 1 leader_uuid: "eae818882c4a4f439ca4948ee1ad0d3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eae818882c4a4f439ca4948ee1ad0d3d" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 44033 } } }
I20260926 00:59:49.178324 5828 sys_catalog.cc:458] T 00000000000000000000000000000000 P eae818882c4a4f439ca4948ee1ad0d3d [sys.catalog]: This master's current role is: LEADER
I20260926 00:59:49.178207 5827 sys_catalog.cc:455] T 00000000000000000000000000000000 P eae818882c4a4f439ca4948ee1ad0d3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eae818882c4a4f439ca4948ee1ad0d3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eae818882c4a4f439ca4948ee1ad0d3d" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 44033 } } }
I20260926 00:59:49.178498 5827 sys_catalog.cc:458] T 00000000000000000000000000000000 P eae818882c4a4f439ca4948ee1ad0d3d [sys.catalog]: This master's current role is: LEADER
I20260926 00:59:49.178546 5835 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 00:59:49.179032 5835 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 00:59:49.180063 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 5755
I20260926 00:59:49.180166 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/master-0/wal/instance
I20260926 00:59:49.180359 3372 external_mini_cluster.cc:1428] Setting key 9458602837ecfc4bc42476b22287f987
I20260926 00:59:49.180649 5835 catalog_manager.cc:1347] Generated new cluster ID: 41a8b7192477476d99ba0a8d074b8782
I20260926 00:59:49.180725 5835 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 00:59:49 dist-test-slave-7v10 krb5kdc[5742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 00:59:49.191854 5846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:49.180783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42170 (local address 127.3.75.62:44033)
0926 00:59:49.181170 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:49.181179 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:49.181210 (+ 31us) server_negotiation.cc:407] Connection header received
0926 00:59:49.181320 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:49.181326 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:49.181379 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:49.181527 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:49.182243 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:49.183720 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:49.184753 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:49.185468 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:49.185604 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:49.188336 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:49.188339 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:49.188340 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 00:59:49.188350 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:49.190213 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:49.190624 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:49.190627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:49.190627 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:49.190724 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:49.190958 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:49.190959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:49.190959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:49.191239 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:49.191399 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:49.191510 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 00:59:49.191633 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":73,"threads_started":1}
I20260926 00:59:49.197389 5835 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 00:59:49.198069 5835 catalog_manager.cc:1504] Loading token signing keys...
I20260926 00:59:49.205884 5835 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eae818882c4a4f439ca4948ee1ad0d3d: Generated new TSK 0
I20260926 00:59:49.206084 5835 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.75.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab.
Entry for principal kudu/127.3.75.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab.
I20260926 00:59:49.231720 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.1:0
--local_ip_for_outbound_sockets=127.3.75.1
--webserver_interface=127.3.75.1
--webserver_port=0
--tserver_master_addrs=127.3.75.62:44033
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab
--principal=kudu/127.3.75.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:37065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 00:59:49.311326 5849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 00:59:49.311525 5849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 00:59:49.311549 5849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 00:59:49.312640 5849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 00:59:49.312674 5849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 00:59:49.312700 5849 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.1
I20260926 00:59:49.314039 5849 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:37065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab
--principal=kudu/127.3.75.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.75.1
--webserver_port=0
--tserver_master_addrs=127.3.75.62:44033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5849
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.1
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 00:59:49.314224 5849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 00:59:49.314393 5849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 00:59:49.319344 5854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:49.319312 5857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 00:59:49.319347 5849 server_base.cc:1034] running on GCE node
W20260926 00:59:49.319319 5855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 00:59:49.319787 5849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 00:59:49.320037 5849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 00:59:49.321197 5849 hybrid_clock.cc:648] HybridClock initialized: now 1790384389321172 us; error 37 us; skew 500 ppm
Sep 26 00:59:49 dist-test-slave-7v10 krb5kdc[5742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384389, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 00:59:49.323621 5849 init.cc:377] Logged in from keytab as kudu/127.3.75.1@KRBTEST.COM (short username kudu)
I20260926 00:59:49.324088 5849 webserver.cc:460] Webserver started at http://127.3.75.1:39945/ using document root <none> and password file <none>
I20260926 00:59:49.324314 5849 fs_manager.cc:359] Metadata directory not provided
I20260926 00:59:49.324380 5849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 00:59:49.324539 5849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 00:59:49.325630 5849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-0/data/instance:
uuid: "152beb26c36e418da9a58b2255c21c28"
format_stamp: "Formatted at 2026-09-26 00:59:49 on dist-test-slave-7v10"
server_key: "285b2c7ff2ca0cde8bee3aaf16956485"
server_key_iv: "239ae7ac8a3a08831cfa397bfbae6926"
server_key_version: "encryptionkey@0"
I20260926 00:59:49.326010 5849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-0/wal/instance:
uuid: "152beb26c36e418da9a58b2255c21c28"
format_stamp: "Formatted at 2026-09-26 00:59:49 on dist-test-slave-7v10"
server_key: "285b2c7ff2ca0cde8bee3aaf16956485"
server_key_iv: "239ae7ac8a3a08831cfa397bfbae6926"
server_key_version: "encryptionkey@0"
I20260926 00:59:49.327395 5849 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 00:59:49.328143 5864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:49.328310 5849 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 00:59:49.328408 5849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-0/wal
uuid: "152beb26c36e418da9a58b2255c21c28"
format_stamp: "Formatted at 2026-09-26 00:59:49 on dist-test-slave-7v10"
server_key: "285b2c7ff2ca0cde8bee3aaf16956485"
server_key_iv: "239ae7ac8a3a08831cfa397bfbae6926"
server_key_version: "encryptionkey@0"
I20260926 00:59:49.328500 5849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 00:59:49.349048 5849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 00:59:49.350992 5849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 00:59:49.351152 5849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 00:59:49.351385 5849 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 00:59:49.351707 5849 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 00:59:49.351763 5849 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:49.351807 5849 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 00:59:49.351835 5849 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:49.357764 5849 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.1:41311
I20260926 00:59:49.357815 5977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.1:41311 every 8 connection(s)
I20260926 00:59:49.358119 5849 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
Sep 26 00:59:49 dist-test-slave-7v10 krb5kdc[5742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384389, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.1@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 00:59:49.366236 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 5849
I20260926 00:59:49.366338 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-0/wal/instance
I20260926 00:59:49.366530 3372 external_mini_cluster.cc:1428] Setting key 02710655d8e026f4a1c410853cbf4eaf
I20260926 00:59:49.366966 5846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:49.358814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:43109 (local address 127.3.75.62:44033)
0926 00:59:49.358914 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:49.358916 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:49.359099 (+ 183us) server_negotiation.cc:407] Connection header received
0926 00:59:49.359549 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:49.359550 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:49.359562 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:49.359611 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:49.360520 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:49.361043 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:49.361995 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:49.362187 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:49.362237 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:49.364320 (+ 2083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:49.364321 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:49.364321 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 00:59:49.364326 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:49.365549 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:49.365896 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:49.365897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:49.365897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:49.365950 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:49.366166 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:49.366167 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:49.366167 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:49.366318 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:49.366389 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:49.366750 (+ 361us) server_negotiation.cc:299] Negotiation successful
0926 00:59:49.366860 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 00:59:49.367182 5980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:49.358776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44033 (local address 127.3.75.1:43109)
0926 00:59:49.358972 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:49.359018 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:49.359469 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:49.359683 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:49.359686 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:49.360261 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0926 00:59:49.360446 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:49.360451 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:49.361124 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:49.361125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:49.361924 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:49.361926 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:49.362258 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:49.362259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:49.362356 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:49.362893 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 00:59:49.362901 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 00:59:49.364240 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 00:59:49.365619 (+ 1379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 00:59:49.365626 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 00:59:49.365627 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 00:59:49.365833 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 00:59:49.366007 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 00:59:49.366007 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 00:59:49.366007 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 00:59:49.366103 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 00:59:49.366427 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 00:59:49.366434 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 00:59:49.366582 (+ 148us) client_negotiation.cc:769] Sending connection context
0926 00:59:49.366723 (+ 141us) client_negotiation.cc:240] Negotiation successful
0926 00:59:49.366946 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":116,"threads_started":1}
I20260926 00:59:49.367861 5978 heartbeater.cc:347] Connected to a master server at 127.3.75.62:44033
I20260926 00:59:49.367972 5978 heartbeater.cc:464] Registering TS with master...
I20260926 00:59:49.368167 5978 heartbeater.cc:511] Master 127.3.75.62:44033 requested a full tablet report, sending...
I20260926 00:59:49.368695 5787 ts_manager.cc:194] Registered new tserver with Master: 152beb26c36e418da9a58b2255c21c28 (127.3.75.1:41311)
I20260926 00:59:49.369612 5787 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.75.1@KRBTEST.COM'} at 127.3.75.1:43109
WARNING: no policy specified for kudu/127.3.75.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab.
Entry for principal kudu/127.3.75.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab.
I20260926 00:59:49.389930 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.2:0
--local_ip_for_outbound_sockets=127.3.75.2
--webserver_interface=127.3.75.2
--webserver_port=0
--tserver_master_addrs=127.3.75.62:44033
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab
--principal=kudu/127.3.75.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:37065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 00:59:49.469345 5983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 00:59:49.469539 5983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 00:59:49.469563 5983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 00:59:49.470752 5983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 00:59:49.470793 5983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 00:59:49.470820 5983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.2
I20260926 00:59:49.472409 5983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:37065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab
--principal=kudu/127.3.75.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.75.2
--webserver_port=0
--tserver_master_addrs=127.3.75.62:44033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.2
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 00:59:49.472641 5983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 00:59:49.472862 5983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 00:59:49.477802 5988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 00:59:49.477834 5983 server_base.cc:1034] running on GCE node
W20260926 00:59:49.477819 5989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:49.477819 5991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 00:59:49.478294 5983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 00:59:49.478559 5983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 00:59:49.479714 5983 hybrid_clock.cc:648] HybridClock initialized: now 1790384389479704 us; error 26 us; skew 500 ppm
Sep 26 00:59:49 dist-test-slave-7v10 krb5kdc[5742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384389, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 00:59:49.482131 5983 init.cc:377] Logged in from keytab as kudu/127.3.75.2@KRBTEST.COM (short username kudu)
I20260926 00:59:49.482546 5983 webserver.cc:460] Webserver started at http://127.3.75.2:38737/ using document root <none> and password file <none>
I20260926 00:59:49.482766 5983 fs_manager.cc:359] Metadata directory not provided
I20260926 00:59:49.482836 5983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 00:59:49.483019 5983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 00:59:49.484042 5983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-1/data/instance:
uuid: "801de4ba742e4303abec08cf9c5152d0"
format_stamp: "Formatted at 2026-09-26 00:59:49 on dist-test-slave-7v10"
server_key: "29f4999cbfc12e9c03e3e3d3f909b93d"
server_key_iv: "11a0900b141e0aaaa250c4c95ac49617"
server_key_version: "encryptionkey@0"
I20260926 00:59:49.484421 5983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-1/wal/instance:
uuid: "801de4ba742e4303abec08cf9c5152d0"
format_stamp: "Formatted at 2026-09-26 00:59:49 on dist-test-slave-7v10"
server_key: "29f4999cbfc12e9c03e3e3d3f909b93d"
server_key_iv: "11a0900b141e0aaaa250c4c95ac49617"
server_key_version: "encryptionkey@0"
I20260926 00:59:49.485683 5983 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 00:59:49.486370 5998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:49.486539 5983 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 00:59:49.486636 5983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-1/wal
uuid: "801de4ba742e4303abec08cf9c5152d0"
format_stamp: "Formatted at 2026-09-26 00:59:49 on dist-test-slave-7v10"
server_key: "29f4999cbfc12e9c03e3e3d3f909b93d"
server_key_iv: "11a0900b141e0aaaa250c4c95ac49617"
server_key_version: "encryptionkey@0"
I20260926 00:59:49.486712 5983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 00:59:49.501176 5983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 00:59:49.503444 5983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 00:59:49.503602 5983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 00:59:49.503819 5983 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 00:59:49.504119 5983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 00:59:49.504172 5983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:49.504212 5983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 00:59:49.504242 5983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:49.509709 5983 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.2:42223
I20260926 00:59:49.509759 6111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.2:42223 every 8 connection(s)
I20260926 00:59:49.510048 5983 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
I20260926 00:59:49.514752 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 5983
I20260926 00:59:49.514891 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-1/wal/instance
I20260926 00:59:49.515079 3372 external_mini_cluster.cc:1428] Setting key 03deb3b695eb04b629c9c9f9d3239317
Sep 26 00:59:49 dist-test-slave-7v10 krb5kdc[5742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384389, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.2@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 00:59:49.519593 6114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:49.510637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44033 (local address 127.3.75.2:48651)
0926 00:59:49.510818 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:49.510859 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:49.511365 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:49.511596 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:49.511599 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:49.512137 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0926 00:59:49.512315 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:49.512319 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:49.512955 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:49.512955 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:49.513715 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:49.513716 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:49.514046 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:49.514046 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:49.514140 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:49.514616 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 00:59:49.514631 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 00:59:49.516350 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 00:59:49.518178 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 00:59:49.518184 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 00:59:49.518186 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 00:59:49.518393 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 00:59:49.518624 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 00:59:49.518625 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 00:59:49.518626 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 00:59:49.518728 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 00:59:49.519042 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 00:59:49.519049 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 00:59:49.519178 (+ 129us) client_negotiation.cc:769] Sending connection context
0926 00:59:49.519302 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 00:59:49.519438 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":86,"threads_started":1}
I20260926 00:59:49.519588 5846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:49.510689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:48651 (local address 127.3.75.62:44033)
0926 00:59:49.510761 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:49.510763 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:49.510967 (+ 204us) server_negotiation.cc:407] Connection header received
0926 00:59:49.511455 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:49.511456 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:49.511466 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:49.511525 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:49.512392 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:49.512874 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:49.513789 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:49.513971 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:49.514020 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:49.516441 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:49.516442 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:49.516442 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 00:59:49.516449 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:49.518093 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:49.518486 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:49.518487 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:49.518488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:49.518552 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:49.518808 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:49.518809 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:49.518809 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:49.518974 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:49.519050 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:49.519357 (+ 307us) server_negotiation.cc:299] Negotiation successful
0926 00:59:49.519464 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 00:59:49.520095 6112 heartbeater.cc:347] Connected to a master server at 127.3.75.62:44033
I20260926 00:59:49.520184 6112 heartbeater.cc:464] Registering TS with master...
I20260926 00:59:49.520370 6112 heartbeater.cc:511] Master 127.3.75.62:44033 requested a full tablet report, sending...
I20260926 00:59:49.520764 5787 ts_manager.cc:194] Registered new tserver with Master: 801de4ba742e4303abec08cf9c5152d0 (127.3.75.2:42223)
I20260926 00:59:49.521144 5787 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.75.2@KRBTEST.COM'} at 127.3.75.2:48651
WARNING: no policy specified for kudu/127.3.75.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab.
Entry for principal kudu/127.3.75.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab.
I20260926 00:59:49.543910 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.3:0
--local_ip_for_outbound_sockets=127.3.75.3
--webserver_interface=127.3.75.3
--webserver_port=0
--tserver_master_addrs=127.3.75.62:44033
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab
--principal=kudu/127.3.75.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:37065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 00:59:49.622536 6117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 00:59:49.622735 6117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 00:59:49.622756 6117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 00:59:49.623957 6117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 00:59:49.624001 6117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 00:59:49.624027 6117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.3
I20260926 00:59:49.625433 6117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:37065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab
--principal=kudu/127.3.75.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.75.3
--webserver_port=0
--tserver_master_addrs=127.3.75.62:44033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.3
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 00:59:49.625667 6117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 00:59:49.625888 6117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 00:59:49.630671 6125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:49.630673 6123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:49.630710 6122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 00:59:49.631022 6117 server_base.cc:1034] running on GCE node
I20260926 00:59:49.631211 6117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 00:59:49.631548 6117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 00:59:49.632706 6117 hybrid_clock.cc:648] HybridClock initialized: now 1790384389632693 us; error 29 us; skew 500 ppm
Sep 26 00:59:49 dist-test-slave-7v10 krb5kdc[5742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384389, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 00:59:49.635180 6117 init.cc:377] Logged in from keytab as kudu/127.3.75.3@KRBTEST.COM (short username kudu)
I20260926 00:59:49.635586 6117 webserver.cc:460] Webserver started at http://127.3.75.3:42331/ using document root <none> and password file <none>
I20260926 00:59:49.635800 6117 fs_manager.cc:359] Metadata directory not provided
I20260926 00:59:49.635844 6117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 00:59:49.636009 6117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 00:59:49.637019 6117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-2/data/instance:
uuid: "2ba65578140949fc8b380c5a2071b40c"
format_stamp: "Formatted at 2026-09-26 00:59:49 on dist-test-slave-7v10"
server_key: "fa5c526da98c6c334ace66f3fa36e83a"
server_key_iv: "2da4978737ed4d1164fc4558332bf484"
server_key_version: "encryptionkey@0"
I20260926 00:59:49.637364 6117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-2/wal/instance:
uuid: "2ba65578140949fc8b380c5a2071b40c"
format_stamp: "Formatted at 2026-09-26 00:59:49 on dist-test-slave-7v10"
server_key: "fa5c526da98c6c334ace66f3fa36e83a"
server_key_iv: "2da4978737ed4d1164fc4558332bf484"
server_key_version: "encryptionkey@0"
I20260926 00:59:49.638671 6117 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 00:59:49.639711 6132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:49.639873 6117 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 00:59:49.639963 6117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-2/wal
uuid: "2ba65578140949fc8b380c5a2071b40c"
format_stamp: "Formatted at 2026-09-26 00:59:49 on dist-test-slave-7v10"
server_key: "fa5c526da98c6c334ace66f3fa36e83a"
server_key_iv: "2da4978737ed4d1164fc4558332bf484"
server_key_version: "encryptionkey@0"
I20260926 00:59:49.640033 6117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 00:59:49.669042 6117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 00:59:49.671401 6117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 00:59:49.671569 6117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 00:59:49.671823 6117 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 00:59:49.672147 6117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 00:59:49.672214 6117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:49.672252 6117 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 00:59:49.672288 6117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:49.678711 6117 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.3:39333
I20260926 00:59:49.678768 6245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.3:39333 every 8 connection(s)
I20260926 00:59:49.679183 6117 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
Sep 26 00:59:49 dist-test-slave-7v10 krb5kdc[5742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384389, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.3@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 00:59:49.687723 5846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:49.679933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.3:42845 (local address 127.3.75.62:44033)
0926 00:59:49.680001 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:49.680003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:49.680181 (+ 178us) server_negotiation.cc:407] Connection header received
0926 00:59:49.680647 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:49.680649 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:49.680661 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:49.680739 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:49.681628 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:49.682118 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:49.683060 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:49.683247 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:49.683343 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:49.685364 (+ 2021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:49.685365 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:49.685365 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 00:59:49.685372 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:49.686500 (+ 1128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:49.686935 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:49.686936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:49.686937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:49.686994 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:49.687223 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:49.687223 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:49.687223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:49.687320 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:49.687369 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:49.687604 (+ 235us) server_negotiation.cc:299] Negotiation successful
0926 00:59:49.687661 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 00:59:49.687928 6248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:49.679896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44033 (local address 127.3.75.3:42845)
0926 00:59:49.680057 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:49.680098 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:49.680560 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:49.680810 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:49.680813 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:49.681359 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0926 00:59:49.681544 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:49.681550 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:49.682200 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:49.682201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:49.682989 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:49.682991 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:49.683355 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:49.683356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:49.683447 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:49.683869 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 00:59:49.683876 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 00:59:49.685256 (+ 1380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 00:59:49.686575 (+ 1319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 00:59:49.686581 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 00:59:49.686583 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 00:59:49.686808 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 00:59:49.687057 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 00:59:49.687058 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 00:59:49.687058 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 00:59:49.687162 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 00:59:49.687381 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 00:59:49.687385 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 00:59:49.687493 (+ 108us) client_negotiation.cc:769] Sending connection context
0926 00:59:49.687580 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 00:59:49.687702 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":84,"threads_started":1}
I20260926 00:59:49.688472 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 6117
I20260926 00:59:49.688493 6246 heartbeater.cc:347] Connected to a master server at 127.3.75.62:44033
I20260926 00:59:49.688582 6246 heartbeater.cc:464] Registering TS with master...
I20260926 00:59:49.688585 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-2/wal/instance
I20260926 00:59:49.688781 6246 heartbeater.cc:511] Master 127.3.75.62:44033 requested a full tablet report, sending...
I20260926 00:59:49.688792 3372 external_mini_cluster.cc:1428] Setting key d076784783a6461960e44cd9d01cc210
I20260926 00:59:49.689272 5786 ts_manager.cc:194] Registered new tserver with Master: 2ba65578140949fc8b380c5a2071b40c (127.3.75.3:39333)
I20260926 00:59:49.689749 3372 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 00:59:49.689787 5786 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.75.3@KRBTEST.COM'} at 127.3.75.3:42845
W20260926 00:59:49.700654 5846 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 00:59:49.699829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42178 (local address 127.3.75.62:44033)
0926 00:59:49.699876 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:49.699878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:49.699952 (+ 74us) server_negotiation.cc:407] Connection header received
0926 00:59:49.700009 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:49.700009 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:49.700018 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:49.700061 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:49.700590 (+ 529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42178: BlockingRecv error: recv got EOF from 127.0.0.1:42178 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 00:59:49.701529 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 5849
I20260926 00:59:49.706750 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 5983
I20260926 00:59:49.711498 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 6117
I20260926 00:59:49.716975 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 5755
2026-09-26T00:59:49Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1801 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384372073861-3372-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 00:59:49 dist-test-slave-7v10 krb5kdc[6258](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 00:59:49 dist-test-slave-7v10 krb5kdc[6258](info): set up 2 sockets
Sep 26 00:59:49 dist-test-slave-7v10 krb5kdc[6258](info): commencing operation
krb5kdc: starting...
W20260926 00:59:50.742030 3372 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Sep 26 00:59:50 dist-test-slave-7v10 krb5kdc[6258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T00:59:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T00:59:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.75.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab.
Entry for principal kudu/127.3.75.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab.
I20260926 00:59:50.857264 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384372073861-3372-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384372073861-3372-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384372073861-3372-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:46591
--webserver_interface=127.3.75.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:36211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.75.62:46591
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 00:59:50.936417 6271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 00:59:50.936652 6271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 00:59:50.936690 6271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 00:59:50.937924 6271 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 00:59:50.937992 6271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 00:59:50.938015 6271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 00:59:50.938037 6271 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 00:59:50.938066 6271 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 00:59:50.939541 6271 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:36211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384372073861-3372-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384372073861-3372-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.75.62:46591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:46591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384372073861-3372-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 00:59:50.939808 6271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 00:59:50.940027 6271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 00:59:50.944757 6276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:50.944793 6277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 00:59:50.944864 6271 server_base.cc:1034] running on GCE node
W20260926 00:59:50.944782 6279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 00:59:50.945257 6271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 00:59:50.945588 6271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 00:59:50.946786 6271 hybrid_clock.cc:648] HybridClock initialized: now 1790384390946771 us; error 45 us; skew 500 ppm
Sep 26 00:59:50 dist-test-slave-7v10 krb5kdc[6258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384390, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 00:59:50.949270 6271 init.cc:377] Logged in from keytab as kudu/127.3.75.62@KRBTEST.COM (short username kudu)
I20260926 00:59:50.949654 6271 webserver.cc:460] Webserver started at http://127.3.75.62:33075/ using document root <none> and password file <none>
I20260926 00:59:50.949831 6271 fs_manager.cc:359] Metadata directory not provided
I20260926 00:59:50.949869 6271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 00:59:50.949954 6271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 00:59:50.951000 6271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384372073861-3372-0/minicluster-data/master-0/data/instance:
uuid: "5ff998dda01348229e9ede507d4409e1"
format_stamp: "Formatted at 2026-09-26 00:59:50 on dist-test-slave-7v10"
server_key: "06ca7494346e06aa786ef3f54205c349"
server_key_iv: "c2965abf084ab0cfb2c0836b605ac340"
server_key_version: "encryptionkey@0"
I20260926 00:59:50.951332 6271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384372073861-3372-0/minicluster-data/master-0/wal/instance:
uuid: "5ff998dda01348229e9ede507d4409e1"
format_stamp: "Formatted at 2026-09-26 00:59:50 on dist-test-slave-7v10"
server_key: "06ca7494346e06aa786ef3f54205c349"
server_key_iv: "c2965abf084ab0cfb2c0836b605ac340"
server_key_version: "encryptionkey@0"
I20260926 00:59:50.952570 6271 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 00:59:50.953233 6286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:50.953418 6271 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 00:59:50.953491 6271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384372073861-3372-0/minicluster-data/master-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384372073861-3372-0/minicluster-data/master-0/wal
uuid: "5ff998dda01348229e9ede507d4409e1"
format_stamp: "Formatted at 2026-09-26 00:59:50 on dist-test-slave-7v10"
server_key: "06ca7494346e06aa786ef3f54205c349"
server_key_iv: "c2965abf084ab0cfb2c0836b605ac340"
server_key_version: "encryptionkey@0"
I20260926 00:59:50.953541 6271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384372073861-3372-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384372073861-3372-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384372073861-3372-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 00:59:50.962273 6271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 00:59:50.963907 6271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 00:59:50.964006 6271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 00:59:50.967667 6271 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.62:46591
I20260926 00:59:50.967694 6338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.62:46591 every 8 connection(s)
I20260926 00:59:50.968011 6271 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
I20260926 00:59:50.968803 6339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 00:59:50.970863 6339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ff998dda01348229e9ede507d4409e1: Bootstrap starting.
I20260926 00:59:50.971532 6339 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ff998dda01348229e9ede507d4409e1: Neither blocks nor log segments found. Creating new log.
I20260926 00:59:50.971728 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 6271
I20260926 00:59:50.971833 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384372073861-3372-0/minicluster-data/master-0/wal/instance
I20260926 00:59:50.971848 6339 log.cc:824] T 00000000000000000000000000000000 P 5ff998dda01348229e9ede507d4409e1: Log is configured to *not* fsync() on all Append() calls
I20260926 00:59:50.972033 3372 external_mini_cluster.cc:1428] Setting key 2ce05ebe1e442c805244d9df682fe963
I20260926 00:59:50.972697 6339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ff998dda01348229e9ede507d4409e1: No bootstrap required, opened a new log
I20260926 00:59:50.975272 6339 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ff998dda01348229e9ede507d4409e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ff998dda01348229e9ede507d4409e1" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 46591 } }
I20260926 00:59:50.975422 6339 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ff998dda01348229e9ede507d4409e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 00:59:50.975471 6339 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ff998dda01348229e9ede507d4409e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff998dda01348229e9ede507d4409e1, State: Initialized, Role: FOLLOWER
I20260926 00:59:50.975590 6339 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ff998dda01348229e9ede507d4409e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ff998dda01348229e9ede507d4409e1" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 46591 } }
I20260926 00:59:50.975665 6339 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ff998dda01348229e9ede507d4409e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 00:59:50.975708 6339 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ff998dda01348229e9ede507d4409e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 00:59:50.975752 6339 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ff998dda01348229e9ede507d4409e1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 00:59:50.976253 6339 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ff998dda01348229e9ede507d4409e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ff998dda01348229e9ede507d4409e1" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 46591 } }
I20260926 00:59:50.976365 6339 leader_election.cc:302] T 00000000000000000000000000000000 P 5ff998dda01348229e9ede507d4409e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ff998dda01348229e9ede507d4409e1; no voters:
I20260926 00:59:50.976509 6339 leader_election.cc:288] T 00000000000000000000000000000000 P 5ff998dda01348229e9ede507d4409e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 00:59:50.976532 6344 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ff998dda01348229e9ede507d4409e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 00:59:50.976709 6344 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ff998dda01348229e9ede507d4409e1 [term 1 LEADER]: Becoming Leader. State: Replica: 5ff998dda01348229e9ede507d4409e1, State: Running, Role: LEADER
I20260926 00:59:50.976825 6339 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ff998dda01348229e9ede507d4409e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 00:59:50.976819 6344 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ff998dda01348229e9ede507d4409e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ff998dda01348229e9ede507d4409e1" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 46591 } }
Sep 26 00:59:50 dist-test-slave-7v10 krb5kdc[6258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 00:59:50.978642 6346 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ff998dda01348229e9ede507d4409e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ff998dda01348229e9ede507d4409e1. Latest consensus state: current_term: 1 leader_uuid: "5ff998dda01348229e9ede507d4409e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ff998dda01348229e9ede507d4409e1" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 46591 } } }
I20260926 00:59:50.978742 6345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ff998dda01348229e9ede507d4409e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ff998dda01348229e9ede507d4409e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ff998dda01348229e9ede507d4409e1" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 46591 } } }
I20260926 00:59:50.978813 6346 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ff998dda01348229e9ede507d4409e1 [sys.catalog]: This master's current role is: LEADER
I20260926 00:59:50.978816 6345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ff998dda01348229e9ede507d4409e1 [sys.catalog]: This master's current role is: LEADER
W20260926 00:59:50.979486 6358 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5ff998dda01348229e9ede507d4409e1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 00:59:50.979555 6358 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 00:59:50.979707 6359 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 00:59:50.979856 6359 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 00:59:50.981285 6359 catalog_manager.cc:1347] Generated new cluster ID: 2ecc3f8ab4cf4c87b1eceb5b9b665438
I20260926 00:59:50.981336 6359 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 00:59:50.982679 6343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:50.972531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38210 (local address 127.3.75.62:46591)
0926 00:59:50.972686 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:50.972694 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:50.972727 (+ 33us) server_negotiation.cc:407] Connection header received
0926 00:59:50.973163 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:50.973169 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:50.973211 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:50.973314 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:50.974098 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:50.975049 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:50.975976 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:50.976410 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:50.976506 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:50.979020 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:50.979024 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:50.979024 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 00:59:50.979035 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:50.981022 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:50.981450 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:50.981453 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:50.981453 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:50.981549 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:50.981780 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:50.981781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:50.981781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:50.982076 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:50.982210 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:50.982327 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 00:59:50.982472 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":83,"threads_started":1}
I20260926 00:59:50.983464 3372 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 00:59:50.998875 6343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:50.994989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38222 (local address 127.3.75.62:46591)
0926 00:59:50.995027 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:50.995028 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:50.995147 (+ 119us) server_negotiation.cc:407] Connection header received
0926 00:59:50.995178 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:50.995178 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:50.995188 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:50.995254 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:50.995610 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:50.996336 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:50.997270 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:50.997587 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:50.997665 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:50.998390 (+ 725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:50.998391 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:50.998391 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 00:59:50.998610 (+ 219us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:50.998628 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 00:59:50.998635 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:50.998693 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:50.998770 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 00:59:50.998796 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 00:59:51.001113 6359 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 00:59:51.001678 6359 catalog_manager.cc:1504] Loading token signing keys...
I20260926 00:59:51.014186 6359 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ff998dda01348229e9ede507d4409e1: Generated new TSK 0
I20260926 00:59:51.014391 6359 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 00:59:51.021093 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 6271
2026-09-26T00:59:51Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1302 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 00:59:51 dist-test-slave-7v10 krb5kdc[6372](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 00:59:51 dist-test-slave-7v10 krb5kdc[6372](info): set up 2 sockets
Sep 26 00:59:51 dist-test-slave-7v10 krb5kdc[6372](info): commencing operation
krb5kdc: starting...
W20260926 00:59:52.065775 3372 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Sep 26 00:59:52 dist-test-slave-7v10 krb5kdc[6372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T00:59:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T00:59:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.75.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab.
Entry for principal kudu/127.3.75.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab.
I20260926 00:59:52.201691 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:38595
--webserver_interface=127.3.75.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:46257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.75.62:38595
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 00:59:52.282541 6385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 00:59:52.282794 6385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 00:59:52.282829 6385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 00:59:52.283993 6385 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 00:59:52.284067 6385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 00:59:52.284091 6385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 00:59:52.284113 6385 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 00:59:52.284143 6385 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 00:59:52.285593 6385 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:46257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.75.62:38595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:38595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6385
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 00:59:52.285857 6385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 00:59:52.286109 6385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 00:59:52.291062 6391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:52.291054 6390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:52.291138 6393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 00:59:52.291368 6385 server_base.cc:1034] running on GCE node
I20260926 00:59:52.291615 6385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 00:59:52.291965 6385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 00:59:52.293154 6385 hybrid_clock.cc:648] HybridClock initialized: now 1790384392293128 us; error 37 us; skew 500 ppm
Sep 26 00:59:52 dist-test-slave-7v10 krb5kdc[6372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384392, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 00:59:52.295713 6385 init.cc:377] Logged in from keytab as kudu/127.3.75.62@KRBTEST.COM (short username kudu)
I20260926 00:59:52.296204 6385 webserver.cc:460] Webserver started at http://127.3.75.62:45917/ using document root <none> and password file <none>
I20260926 00:59:52.296402 6385 fs_manager.cc:359] Metadata directory not provided
I20260926 00:59:52.296445 6385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 00:59:52.296532 6385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 00:59:52.297551 6385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/master-0/data/instance:
uuid: "c448cc29ae424c5bb9a9657824e95562"
format_stamp: "Formatted at 2026-09-26 00:59:52 on dist-test-slave-7v10"
server_key: "3e6721fe50878f8346f41529cca86068"
server_key_iv: "cd46edbf2f6d9633ac1ca46b8877d7b2"
server_key_version: "encryptionkey@0"
I20260926 00:59:52.297859 6385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/master-0/wal/instance:
uuid: "c448cc29ae424c5bb9a9657824e95562"
format_stamp: "Formatted at 2026-09-26 00:59:52 on dist-test-slave-7v10"
server_key: "3e6721fe50878f8346f41529cca86068"
server_key_iv: "cd46edbf2f6d9633ac1ca46b8877d7b2"
server_key_version: "encryptionkey@0"
I20260926 00:59:52.299188 6385 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 00:59:52.299857 6400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:52.300009 6385 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 00:59:52.300091 6385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/master-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/master-0/wal
uuid: "c448cc29ae424c5bb9a9657824e95562"
format_stamp: "Formatted at 2026-09-26 00:59:52 on dist-test-slave-7v10"
server_key: "3e6721fe50878f8346f41529cca86068"
server_key_iv: "cd46edbf2f6d9633ac1ca46b8877d7b2"
server_key_version: "encryptionkey@0"
I20260926 00:59:52.300141 6385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 00:59:52.321408 6385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 00:59:52.323518 6385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 00:59:52.323622 6385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 00:59:52.327361 6385 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.62:38595
I20260926 00:59:52.327426 6452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.62:38595 every 8 connection(s)
I20260926 00:59:52.327760 6385 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
I20260926 00:59:52.328776 6453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 00:59:52.331055 6453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c448cc29ae424c5bb9a9657824e95562: Bootstrap starting.
I20260926 00:59:52.331609 6453 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c448cc29ae424c5bb9a9657824e95562: Neither blocks nor log segments found. Creating new log.
I20260926 00:59:52.331890 6453 log.cc:824] T 00000000000000000000000000000000 P c448cc29ae424c5bb9a9657824e95562: Log is configured to *not* fsync() on all Append() calls
I20260926 00:59:52.332567 6453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c448cc29ae424c5bb9a9657824e95562: No bootstrap required, opened a new log
I20260926 00:59:52.334131 6453 raft_consensus.cc:348] T 00000000000000000000000000000000 P c448cc29ae424c5bb9a9657824e95562 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c448cc29ae424c5bb9a9657824e95562" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 38595 } }
I20260926 00:59:52.334287 6453 raft_consensus.cc:374] T 00000000000000000000000000000000 P c448cc29ae424c5bb9a9657824e95562 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 00:59:52.334335 6453 raft_consensus.cc:729] T 00000000000000000000000000000000 P c448cc29ae424c5bb9a9657824e95562 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c448cc29ae424c5bb9a9657824e95562, State: Initialized, Role: FOLLOWER
I20260926 00:59:52.334476 6453 consensus_queue.cc:260] T 00000000000000000000000000000000 P c448cc29ae424c5bb9a9657824e95562 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c448cc29ae424c5bb9a9657824e95562" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 38595 } }
I20260926 00:59:52.334558 6453 raft_consensus.cc:388] T 00000000000000000000000000000000 P c448cc29ae424c5bb9a9657824e95562 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 00:59:52.334602 6453 raft_consensus.cc:482] T 00000000000000000000000000000000 P c448cc29ae424c5bb9a9657824e95562 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 00:59:52.334654 6453 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c448cc29ae424c5bb9a9657824e95562 [term 0 FOLLOWER]: Advancing to term 1
I20260926 00:59:52.335511 6453 raft_consensus.cc:504] T 00000000000000000000000000000000 P c448cc29ae424c5bb9a9657824e95562 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c448cc29ae424c5bb9a9657824e95562" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 38595 } }
I20260926 00:59:52.335630 6453 leader_election.cc:302] T 00000000000000000000000000000000 P c448cc29ae424c5bb9a9657824e95562 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c448cc29ae424c5bb9a9657824e95562; no voters:
I20260926 00:59:52.335815 6453 leader_election.cc:288] T 00000000000000000000000000000000 P c448cc29ae424c5bb9a9657824e95562 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 00:59:52.335841 6456 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c448cc29ae424c5bb9a9657824e95562 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 00:59:52.335999 6456 raft_consensus.cc:686] T 00000000000000000000000000000000 P c448cc29ae424c5bb9a9657824e95562 [term 1 LEADER]: Becoming Leader. State: Replica: c448cc29ae424c5bb9a9657824e95562, State: Running, Role: LEADER
I20260926 00:59:52.336076 6453 sys_catalog.cc:564] T 00000000000000000000000000000000 P c448cc29ae424c5bb9a9657824e95562 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 00:59:52.336344 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 6385
I20260926 00:59:52.336447 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/master-0/wal/instance
I20260926 00:59:52.336642 3372 external_mini_cluster.cc:1428] Setting key 144d0bd47aada5a96cde3f03e6824a42
I20260926 00:59:52.336144 6456 consensus_queue.cc:237] T 00000000000000000000000000000000 P c448cc29ae424c5bb9a9657824e95562 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c448cc29ae424c5bb9a9657824e95562" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 38595 } }
I20260926 00:59:52.337344 6457 sys_catalog.cc:455] T 00000000000000000000000000000000 P c448cc29ae424c5bb9a9657824e95562 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c448cc29ae424c5bb9a9657824e95562" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c448cc29ae424c5bb9a9657824e95562" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 38595 } } }
I20260926 00:59:52.337345 6458 sys_catalog.cc:455] T 00000000000000000000000000000000 P c448cc29ae424c5bb9a9657824e95562 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c448cc29ae424c5bb9a9657824e95562. Latest consensus state: current_term: 1 leader_uuid: "c448cc29ae424c5bb9a9657824e95562" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c448cc29ae424c5bb9a9657824e95562" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 38595 } } }
I20260926 00:59:52.337702 6457 sys_catalog.cc:458] T 00000000000000000000000000000000 P c448cc29ae424c5bb9a9657824e95562 [sys.catalog]: This master's current role is: LEADER
I20260926 00:59:52.337709 6458 sys_catalog.cc:458] T 00000000000000000000000000000000 P c448cc29ae424c5bb9a9657824e95562 [sys.catalog]: This master's current role is: LEADER
W20260926 00:59:52.338186 6471 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c448cc29ae424c5bb9a9657824e95562: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 00:59:52.338402 6471 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 00:59:52.338521 6473 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 00:59:52.338703 6473 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 00:59:52.340306 6473 catalog_manager.cc:1347] Generated new cluster ID: c0bbeedb65b74245a7c33556eccc5f3e
I20260926 00:59:52.340360 6473 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 00:59:52 dist-test-slave-7v10 krb5kdc[6372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 00:59:52.347815 6472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:52.337312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46882 (local address 127.3.75.62:38595)
0926 00:59:52.337553 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:52.337585 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:52.337686 (+ 101us) server_negotiation.cc:407] Connection header received
0926 00:59:52.338086 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:52.338095 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:52.338155 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:52.338370 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:52.339173 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:52.340445 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:52.341195 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:52.341612 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:52.341737 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:52.344087 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:52.344090 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:52.344091 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 00:59:52.344110 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:52.346264 (+ 2154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:52.346643 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:52.346646 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:52.346646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:52.346746 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:52.346989 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:52.346990 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:52.346990 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:52.347219 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:52.347387 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:52.347484 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 00:59:52.347626 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":70,"threads_started":1}
I20260926 00:59:52.363178 6473 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 00:59:52.363718 6473 catalog_manager.cc:1504] Loading token signing keys...
I20260926 00:59:52.365433 6473 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c448cc29ae424c5bb9a9657824e95562: Generated new TSK 0
I20260926 00:59:52.365617 6473 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.75.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab.
Entry for principal kudu/127.3.75.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab.
I20260926 00:59:52.397372 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.1:0
--local_ip_for_outbound_sockets=127.3.75.1
--webserver_interface=127.3.75.1
--webserver_port=0
--tserver_master_addrs=127.3.75.62:38595
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab
--principal=kudu/127.3.75.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:46257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 00:59:52.484921 6479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 00:59:52.485116 6479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 00:59:52.485139 6479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 00:59:52.486300 6479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 00:59:52.486341 6479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 00:59:52.486367 6479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.1
I20260926 00:59:52.487829 6479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:46257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab
--principal=kudu/127.3.75.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.75.1
--webserver_port=0
--tserver_master_addrs=127.3.75.62:38595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.1
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 00:59:52.488068 6479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 00:59:52.488310 6479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 00:59:52.493232 6484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:52.493202 6485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:52.493290 6487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 00:59:52.493338 6479 server_base.cc:1034] running on GCE node
I20260926 00:59:52.493659 6479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 00:59:52.493901 6479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 00:59:52.495064 6479 hybrid_clock.cc:648] HybridClock initialized: now 1790384392495055 us; error 26 us; skew 500 ppm
Sep 26 00:59:52 dist-test-slave-7v10 krb5kdc[6372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384392, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 00:59:52.497452 6479 init.cc:377] Logged in from keytab as kudu/127.3.75.1@KRBTEST.COM (short username kudu)
I20260926 00:59:52.497925 6479 webserver.cc:460] Webserver started at http://127.3.75.1:38991/ using document root <none> and password file <none>
I20260926 00:59:52.498157 6479 fs_manager.cc:359] Metadata directory not provided
I20260926 00:59:52.498222 6479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 00:59:52.498354 6479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 00:59:52.499440 6479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-0/data/instance:
uuid: "2baf204b7969496792a5e9ada8bc9388"
format_stamp: "Formatted at 2026-09-26 00:59:52 on dist-test-slave-7v10"
server_key: "ee1022db67e4c0316bcabf3e2b960b33"
server_key_iv: "ca2400de3d51b45bac9f8bd90c8fec26"
server_key_version: "encryptionkey@0"
I20260926 00:59:52.499794 6479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-0/wal/instance:
uuid: "2baf204b7969496792a5e9ada8bc9388"
format_stamp: "Formatted at 2026-09-26 00:59:52 on dist-test-slave-7v10"
server_key: "ee1022db67e4c0316bcabf3e2b960b33"
server_key_iv: "ca2400de3d51b45bac9f8bd90c8fec26"
server_key_version: "encryptionkey@0"
I20260926 00:59:52.501180 6479 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 00:59:52.501888 6494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:52.502072 6479 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 00:59:52.502171 6479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-0/wal
uuid: "2baf204b7969496792a5e9ada8bc9388"
format_stamp: "Formatted at 2026-09-26 00:59:52 on dist-test-slave-7v10"
server_key: "ee1022db67e4c0316bcabf3e2b960b33"
server_key_iv: "ca2400de3d51b45bac9f8bd90c8fec26"
server_key_version: "encryptionkey@0"
I20260926 00:59:52.502252 6479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 00:59:52.511207 6479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 00:59:52.513072 6479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 00:59:52.513237 6479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 00:59:52.513471 6479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 00:59:52.513777 6479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 00:59:52.513830 6479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:52.513871 6479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 00:59:52.513909 6479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:52.519618 6479 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.1:46071
I20260926 00:59:52.519660 6607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.1:46071 every 8 connection(s)
I20260926 00:59:52.519970 6479 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
I20260926 00:59:52.523000 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 6479
I20260926 00:59:52.523083 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-0/wal/instance
I20260926 00:59:52.523284 3372 external_mini_cluster.cc:1428] Setting key c43a08f14dceea1b41e0951401bc2119
Sep 26 00:59:52 dist-test-slave-7v10 krb5kdc[6372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384392, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.1@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 00:59:52.529853 6472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:52.520599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:45405 (local address 127.3.75.62:38595)
0926 00:59:52.520642 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:52.520644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:52.520842 (+ 198us) server_negotiation.cc:407] Connection header received
0926 00:59:52.521353 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:52.521354 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:52.521366 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:52.521433 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:52.522338 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:52.522829 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:52.524278 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:52.524508 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:52.524588 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:52.527035 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:52.527037 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:52.527037 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 00:59:52.527045 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:52.528453 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:52.528863 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:52.528864 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:52.528865 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:52.528926 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:52.529191 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:52.529192 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:52.529192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:52.529311 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:52.529382 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:52.529671 (+ 289us) server_negotiation.cc:299] Negotiation successful
0926 00:59:52.529761 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 00:59:52.530089 6610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:52.520552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:38595 (local address 127.3.75.1:45405)
0926 00:59:52.520706 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:52.520747 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:52.521264 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:52.521508 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:52.521511 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:52.522066 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0926 00:59:52.522258 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:52.522262 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:52.522981 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:52.522982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:52.524186 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:52.524189 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:52.524597 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:52.524598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:52.524733 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:52.525292 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 00:59:52.525303 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 00:59:52.526932 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 00:59:52.528543 (+ 1611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 00:59:52.528551 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 00:59:52.528554 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 00:59:52.528785 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 00:59:52.528996 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 00:59:52.528997 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 00:59:52.528998 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 00:59:52.529115 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 00:59:52.529385 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 00:59:52.529393 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 00:59:52.529537 (+ 144us) client_negotiation.cc:769] Sending connection context
0926 00:59:52.529649 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 00:59:52.529813 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":86,"threads_started":1}
I20260926 00:59:52.530805 6608 heartbeater.cc:347] Connected to a master server at 127.3.75.62:38595
I20260926 00:59:52.530946 6608 heartbeater.cc:464] Registering TS with master...
I20260926 00:59:52.531152 6608 heartbeater.cc:511] Master 127.3.75.62:38595 requested a full tablet report, sending...
I20260926 00:59:52.531730 6417 ts_manager.cc:194] Registered new tserver with Master: 2baf204b7969496792a5e9ada8bc9388 (127.3.75.1:46071)
WARNING: no policy specified for kudu/127.3.75.2@KRBTEST.COM; defaulting to no policy
I20260926 00:59:52.532414 6417 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.75.1@KRBTEST.COM'} at 127.3.75.1:45405
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab.
Entry for principal kudu/127.3.75.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab.
I20260926 00:59:52.547771 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.2:0
--local_ip_for_outbound_sockets=127.3.75.2
--webserver_interface=127.3.75.2
--webserver_port=0
--tserver_master_addrs=127.3.75.62:38595
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab
--principal=kudu/127.3.75.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:46257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 00:59:52.626798 6613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 00:59:52.627031 6613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 00:59:52.627054 6613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 00:59:52.628304 6613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 00:59:52.628356 6613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 00:59:52.628384 6613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.2
I20260926 00:59:52.629810 6613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:46257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab
--principal=kudu/127.3.75.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.75.2
--webserver_port=0
--tserver_master_addrs=127.3.75.62:38595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.2
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 00:59:52.630045 6613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 00:59:52.630263 6613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 00:59:52.635109 6621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 00:59:52.635246 6613 server_base.cc:1034] running on GCE node
W20260926 00:59:52.635211 6618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:52.635115 6619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 00:59:52.635558 6613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 00:59:52.635757 6613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 00:59:52.636894 6613 hybrid_clock.cc:648] HybridClock initialized: now 1790384392636870 us; error 40 us; skew 500 ppm
Sep 26 00:59:52 dist-test-slave-7v10 krb5kdc[6372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384392, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 00:59:52.639430 6613 init.cc:377] Logged in from keytab as kudu/127.3.75.2@KRBTEST.COM (short username kudu)
I20260926 00:59:52.639884 6613 webserver.cc:460] Webserver started at http://127.3.75.2:40777/ using document root <none> and password file <none>
I20260926 00:59:52.640097 6613 fs_manager.cc:359] Metadata directory not provided
I20260926 00:59:52.640141 6613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 00:59:52.640275 6613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 00:59:52.641291 6613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-1/data/instance:
uuid: "7245f99e7f104698874698c4dc8b865a"
format_stamp: "Formatted at 2026-09-26 00:59:52 on dist-test-slave-7v10"
server_key: "f193171ef1863c211364c27d58466b80"
server_key_iv: "ae020abde99a45606547223bbc48c353"
server_key_version: "encryptionkey@0"
I20260926 00:59:52.641649 6613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-1/wal/instance:
uuid: "7245f99e7f104698874698c4dc8b865a"
format_stamp: "Formatted at 2026-09-26 00:59:52 on dist-test-slave-7v10"
server_key: "f193171ef1863c211364c27d58466b80"
server_key_iv: "ae020abde99a45606547223bbc48c353"
server_key_version: "encryptionkey@0"
I20260926 00:59:52.642979 6613 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 00:59:52.643698 6628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:52.643893 6613 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 00:59:52.643980 6613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-1/wal
uuid: "7245f99e7f104698874698c4dc8b865a"
format_stamp: "Formatted at 2026-09-26 00:59:52 on dist-test-slave-7v10"
server_key: "f193171ef1863c211364c27d58466b80"
server_key_iv: "ae020abde99a45606547223bbc48c353"
server_key_version: "encryptionkey@0"
I20260926 00:59:52.644054 6613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 00:59:52.659341 6613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 00:59:52.661655 6613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 00:59:52.661818 6613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 00:59:52.662103 6613 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 00:59:52.662415 6613 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 00:59:52.662449 6613 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:52.662472 6613 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 00:59:52.662519 6613 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:52.668181 6613 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.2:40567
I20260926 00:59:52.668229 6741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.2:40567 every 8 connection(s)
I20260926 00:59:52.668543 6613 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
I20260926 00:59:52.672323 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 6613
I20260926 00:59:52.672438 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-1/wal/instance
I20260926 00:59:52.672637 3372 external_mini_cluster.cc:1428] Setting key dbb93d34dbac160b394ee857726c41aa
Sep 26 00:59:52 dist-test-slave-7v10 krb5kdc[6372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384392, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.2@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 00:59:52.678273 6472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:52.669254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:55761 (local address 127.3.75.62:38595)
0926 00:59:52.669318 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:52.669320 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:52.669498 (+ 178us) server_negotiation.cc:407] Connection header received
0926 00:59:52.669964 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:52.669965 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:52.669974 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:52.670034 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:52.670905 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:52.671390 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:52.672513 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:52.672783 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:52.672866 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:52.675464 (+ 2598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:52.675466 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:52.675466 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 00:59:52.675472 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:52.676861 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:52.677341 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:52.677342 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:52.677343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:52.677397 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:52.677673 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:52.677673 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:52.677674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:52.677765 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:52.677810 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:52.678151 (+ 341us) server_negotiation.cc:299] Negotiation successful
0926 00:59:52.678213 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 00:59:52.678610 6744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:52.669211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:38595 (local address 127.3.75.2:55761)
0926 00:59:52.669368 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:52.669409 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:52.669874 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:52.670085 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:52.670088 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:52.670625 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0926 00:59:52.670811 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:52.670815 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:52.671472 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:52.671472 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:52.672431 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:52.672434 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:52.672897 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:52.672898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:52.673049 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:52.673572 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 00:59:52.673583 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 00:59:52.675363 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 00:59:52.676943 (+ 1580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 00:59:52.676950 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 00:59:52.676953 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 00:59:52.677265 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 00:59:52.677458 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 00:59:52.677459 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 00:59:52.677460 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 00:59:52.677602 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 00:59:52.677827 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 00:59:52.677833 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 00:59:52.678014 (+ 181us) client_negotiation.cc:769] Sending connection context
0926 00:59:52.678140 (+ 126us) client_negotiation.cc:240] Negotiation successful
0926 00:59:52.678318 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":92,"threads_started":1}
I20260926 00:59:52.679208 6742 heartbeater.cc:347] Connected to a master server at 127.3.75.62:38595
I20260926 00:59:52.679297 6742 heartbeater.cc:464] Registering TS with master...
I20260926 00:59:52.679464 6742 heartbeater.cc:511] Master 127.3.75.62:38595 requested a full tablet report, sending...
I20260926 00:59:52.679886 6417 ts_manager.cc:194] Registered new tserver with Master: 7245f99e7f104698874698c4dc8b865a (127.3.75.2:40567)
I20260926 00:59:52.680246 6417 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.75.2@KRBTEST.COM'} at 127.3.75.2:55761
WARNING: no policy specified for kudu/127.3.75.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab.
Entry for principal kudu/127.3.75.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab.
I20260926 00:59:52.697337 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.3:0
--local_ip_for_outbound_sockets=127.3.75.3
--webserver_interface=127.3.75.3
--webserver_port=0
--tserver_master_addrs=127.3.75.62:38595
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab
--principal=kudu/127.3.75.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:46257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 00:59:52.778551 6747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 00:59:52.778738 6747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 00:59:52.778759 6747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 00:59:52.779978 6747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 00:59:52.780019 6747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 00:59:52.780045 6747 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.3
I20260926 00:59:52.781433 6747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:46257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab
--principal=kudu/127.3.75.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.75.3
--webserver_port=0
--tserver_master_addrs=127.3.75.62:38595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.3
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 00:59:52.781629 6747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 00:59:52.781797 6747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 00:59:52.786597 6755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:52.786589 6752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:52.786669 6753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 00:59:52.786706 6747 server_base.cc:1034] running on GCE node
I20260926 00:59:52.787046 6747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 00:59:52.787328 6747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 00:59:52.788483 6747 hybrid_clock.cc:648] HybridClock initialized: now 1790384392788480 us; error 34 us; skew 500 ppm
Sep 26 00:59:52 dist-test-slave-7v10 krb5kdc[6372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384392, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 00:59:52.790822 6747 init.cc:377] Logged in from keytab as kudu/127.3.75.3@KRBTEST.COM (short username kudu)
I20260926 00:59:52.791311 6747 webserver.cc:460] Webserver started at http://127.3.75.3:35499/ using document root <none> and password file <none>
I20260926 00:59:52.791570 6747 fs_manager.cc:359] Metadata directory not provided
I20260926 00:59:52.791635 6747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 00:59:52.791785 6747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 00:59:52.792769 6747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-2/data/instance:
uuid: "31f0df17f4c0415a97f453dbcb2427b4"
format_stamp: "Formatted at 2026-09-26 00:59:52 on dist-test-slave-7v10"
server_key: "43ad4198e4e33070b1748cd7adb40acb"
server_key_iv: "f8510b65bb091af5b90e484b0a98fb59"
server_key_version: "encryptionkey@0"
I20260926 00:59:52.793124 6747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-2/wal/instance:
uuid: "31f0df17f4c0415a97f453dbcb2427b4"
format_stamp: "Formatted at 2026-09-26 00:59:52 on dist-test-slave-7v10"
server_key: "43ad4198e4e33070b1748cd7adb40acb"
server_key_iv: "f8510b65bb091af5b90e484b0a98fb59"
server_key_version: "encryptionkey@0"
I20260926 00:59:52.794381 6747 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 00:59:52.795138 6762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:52.795359 6747 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 00:59:52.795471 6747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-2/wal
uuid: "31f0df17f4c0415a97f453dbcb2427b4"
format_stamp: "Formatted at 2026-09-26 00:59:52 on dist-test-slave-7v10"
server_key: "43ad4198e4e33070b1748cd7adb40acb"
server_key_iv: "f8510b65bb091af5b90e484b0a98fb59"
server_key_version: "encryptionkey@0"
I20260926 00:59:52.795552 6747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 00:59:52.812245 6747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 00:59:52.813932 6747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 00:59:52.814092 6747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 00:59:52.814314 6747 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 00:59:52.814622 6747 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 00:59:52.814674 6747 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:52.814706 6747 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 00:59:52.814734 6747 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:52.820376 6747 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.3:46465
I20260926 00:59:52.820417 6875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.3:46465 every 8 connection(s)
I20260926 00:59:52.820730 6747 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
I20260926 00:59:52.822070 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 6747
I20260926 00:59:52.822148 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384372073861-3372-0/minicluster-data/ts-2/wal/instance
I20260926 00:59:52.822364 3372 external_mini_cluster.cc:1428] Setting key 69876bb2cec91a5a9b5ea6fd879e20e1
Sep 26 00:59:52 dist-test-slave-7v10 krb5kdc[6372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384392, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.3@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 00:59:52.829341 6472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:52.821436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.3:33633 (local address 127.3.75.62:38595)
0926 00:59:52.821515 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:52.821517 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:52.821683 (+ 166us) server_negotiation.cc:407] Connection header received
0926 00:59:52.822222 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:52.822223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:52.822238 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:52.822316 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:52.823459 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:52.823943 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:52.824865 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:52.825051 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:52.825103 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:52.827132 (+ 2029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:52.827133 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:52.827134 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 00:59:52.827140 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:52.828260 (+ 1120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:52.828619 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:52.828620 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:52.828620 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:52.828668 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:52.828858 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:52.828859 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:52.828859 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:52.828948 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:52.828994 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:52.829223 (+ 229us) server_negotiation.cc:299] Negotiation successful
0926 00:59:52.829280 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 00:59:52.829557 6878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:52.821407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:38595 (local address 127.3.75.3:33633)
0926 00:59:52.821566 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:52.821599 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:52.822088 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:52.822373 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:52.822383 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:52.823146 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0926 00:59:52.823373 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:52.823384 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:52.824018 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:52.824019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:52.824798 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:52.824800 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:52.825122 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:52.825123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:52.825225 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:52.825726 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 00:59:52.825734 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 00:59:52.827056 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 00:59:52.828326 (+ 1270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 00:59:52.828332 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 00:59:52.828333 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 00:59:52.828544 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 00:59:52.828721 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 00:59:52.828722 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 00:59:52.828722 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 00:59:52.828809 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 00:59:52.828998 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 00:59:52.829001 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 00:59:52.829102 (+ 101us) client_negotiation.cc:769] Sending connection context
0926 00:59:52.829198 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 00:59:52.829306 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":73,"threads_started":1}
I20260926 00:59:52.830065 6876 heartbeater.cc:347] Connected to a master server at 127.3.75.62:38595
I20260926 00:59:52.830166 6876 heartbeater.cc:464] Registering TS with master...
I20260926 00:59:52.830332 6876 heartbeater.cc:511] Master 127.3.75.62:38595 requested a full tablet report, sending...
I20260926 00:59:52.830753 6417 ts_manager.cc:194] Registered new tserver with Master: 31f0df17f4c0415a97f453dbcb2427b4 (127.3.75.3:46465)
I20260926 00:59:52.831147 6417 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.75.3@KRBTEST.COM'} at 127.3.75.3:33633
I20260926 00:59:52.834013 3372 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 00:59:52 dist-test-slave-7v10 krb5kdc[6372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384392, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 26 00:59:52 dist-test-slave-7v10 krb5kdc[6372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384392, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 00:59:52.856007 6472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:52.849358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46888 (local address 127.3.75.62:38595)
0926 00:59:52.849425 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:52.849427 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:52.849585 (+ 158us) server_negotiation.cc:407] Connection header received
0926 00:59:52.849639 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:52.849640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:52.849649 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:52.849723 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:52.850232 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:52.850708 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:52.851433 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:52.851612 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:52.851664 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:52.853703 (+ 2039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:52.853705 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:52.853705 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 00:59:52.853711 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:52.855146 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:52.855430 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:52.855431 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:52.855431 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:52.855479 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:52.855620 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:52.855620 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:52.855621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:52.855747 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:52.855812 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:52.855892 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 00:59:52.855944 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260926 00:59:52.856246 6417 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:46888
I20260926 00:59:52.856717 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 6479
I20260926 00:59:52.861133 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 6613
I20260926 00:59:52.865615 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 6747
I20260926 00:59:52.870263 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 6385
2026-09-26T00:59:52Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1850 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 00:59:52 dist-test-slave-7v10 krb5kdc[6888](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 00:59:52 dist-test-slave-7v10 krb5kdc[6888](info): set up 2 sockets
Sep 26 00:59:52 dist-test-slave-7v10 krb5kdc[6888](info): commencing operation
krb5kdc: starting...
W20260926 00:59:53.906062 3372 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Sep 26 00:59:53 dist-test-slave-7v10 krb5kdc[6888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T00:59:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T00:59:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.75.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab.
Entry for principal kudu/127.3.75.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab.
I20260926 00:59:54.019570 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:35839
--webserver_interface=127.3.75.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:40503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.75.62:35839
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 00:59:54.098111 6901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 00:59:54.098306 6901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 00:59:54.098328 6901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 00:59:54.099401 6901 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 00:59:54.099435 6901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 00:59:54.099449 6901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 00:59:54.099462 6901 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 00:59:54.099474 6901 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 00:59:54.100907 6901 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:40503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.75.62:35839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:35839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 00:59:54.101112 6901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 00:59:54.101282 6901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 00:59:54.106144 6909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:54.106144 6906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 00:59:54.106206 6901 server_base.cc:1034] running on GCE node
W20260926 00:59:54.106168 6907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 00:59:54.106623 6901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 00:59:54.107015 6901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 00:59:54.108177 6901 hybrid_clock.cc:648] HybridClock initialized: now 1790384394108154 us; error 38 us; skew 500 ppm
Sep 26 00:59:54 dist-test-slave-7v10 krb5kdc[6888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384394, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 00:59:54.116499 6901 init.cc:377] Logged in from keytab as kudu/127.3.75.62@KRBTEST.COM (short username kudu)
I20260926 00:59:54.117049 6901 webserver.cc:460] Webserver started at http://127.3.75.62:44965/ using document root <none> and password file <none>
I20260926 00:59:54.117282 6901 fs_manager.cc:359] Metadata directory not provided
I20260926 00:59:54.117352 6901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 00:59:54.117508 6901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 00:59:54.118585 6901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/master-0/data/instance:
uuid: "8a6ef18ce5e544f7a373b7ac86f40fde"
format_stamp: "Formatted at 2026-09-26 00:59:54 on dist-test-slave-7v10"
server_key: "0aa6ab5c7fffdf691ccdbf69cf5aaed1"
server_key_iv: "450649b9f5e01b17f84582e5a802fa6f"
server_key_version: "encryptionkey@0"
I20260926 00:59:54.119016 6901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/master-0/wal/instance:
uuid: "8a6ef18ce5e544f7a373b7ac86f40fde"
format_stamp: "Formatted at 2026-09-26 00:59:54 on dist-test-slave-7v10"
server_key: "0aa6ab5c7fffdf691ccdbf69cf5aaed1"
server_key_iv: "450649b9f5e01b17f84582e5a802fa6f"
server_key_version: "encryptionkey@0"
I20260926 00:59:54.120437 6901 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 00:59:54.121439 6916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:54.121616 6901 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 00:59:54.121697 6901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/master-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/master-0/wal
uuid: "8a6ef18ce5e544f7a373b7ac86f40fde"
format_stamp: "Formatted at 2026-09-26 00:59:54 on dist-test-slave-7v10"
server_key: "0aa6ab5c7fffdf691ccdbf69cf5aaed1"
server_key_iv: "450649b9f5e01b17f84582e5a802fa6f"
server_key_version: "encryptionkey@0"
I20260926 00:59:54.121748 6901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 00:59:54.130712 6901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 00:59:54.132900 6901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 00:59:54.132997 6901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 00:59:54.136747 6901 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.62:35839
I20260926 00:59:54.136778 6968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.62:35839 every 8 connection(s)
I20260926 00:59:54.137253 6901 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
I20260926 00:59:54.137878 6969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 00:59:54.140141 6969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a6ef18ce5e544f7a373b7ac86f40fde: Bootstrap starting.
I20260926 00:59:54.140731 6969 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a6ef18ce5e544f7a373b7ac86f40fde: Neither blocks nor log segments found. Creating new log.
I20260926 00:59:54.141040 6969 log.cc:824] T 00000000000000000000000000000000 P 8a6ef18ce5e544f7a373b7ac86f40fde: Log is configured to *not* fsync() on all Append() calls
I20260926 00:59:54.141705 6969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a6ef18ce5e544f7a373b7ac86f40fde: No bootstrap required, opened a new log
I20260926 00:59:54.143257 6969 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a6ef18ce5e544f7a373b7ac86f40fde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a6ef18ce5e544f7a373b7ac86f40fde" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 35839 } }
I20260926 00:59:54.143402 6969 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a6ef18ce5e544f7a373b7ac86f40fde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 00:59:54.143450 6969 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a6ef18ce5e544f7a373b7ac86f40fde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a6ef18ce5e544f7a373b7ac86f40fde, State: Initialized, Role: FOLLOWER
I20260926 00:59:54.143596 6969 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a6ef18ce5e544f7a373b7ac86f40fde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a6ef18ce5e544f7a373b7ac86f40fde" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 35839 } }
I20260926 00:59:54.143656 6969 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a6ef18ce5e544f7a373b7ac86f40fde [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 00:59:54.143698 6969 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a6ef18ce5e544f7a373b7ac86f40fde [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 00:59:54.143746 6969 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a6ef18ce5e544f7a373b7ac86f40fde [term 0 FOLLOWER]: Advancing to term 1
I20260926 00:59:54.144292 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 6901
I20260926 00:59:54.144416 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/master-0/wal/instance
I20260926 00:59:54.144603 3372 external_mini_cluster.cc:1428] Setting key 208c817655d5f54336e79543e57084fb
I20260926 00:59:54.144742 6969 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a6ef18ce5e544f7a373b7ac86f40fde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a6ef18ce5e544f7a373b7ac86f40fde" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 35839 } }
I20260926 00:59:54.144855 6969 leader_election.cc:302] T 00000000000000000000000000000000 P 8a6ef18ce5e544f7a373b7ac86f40fde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a6ef18ce5e544f7a373b7ac86f40fde; no voters:
I20260926 00:59:54.145025 6969 leader_election.cc:288] T 00000000000000000000000000000000 P 8a6ef18ce5e544f7a373b7ac86f40fde [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 00:59:54.145056 6972 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a6ef18ce5e544f7a373b7ac86f40fde [term 1 FOLLOWER]: Leader election won for term 1
I20260926 00:59:54.145193 6972 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a6ef18ce5e544f7a373b7ac86f40fde [term 1 LEADER]: Becoming Leader. State: Replica: 8a6ef18ce5e544f7a373b7ac86f40fde, State: Running, Role: LEADER
I20260926 00:59:54.145298 6969 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a6ef18ce5e544f7a373b7ac86f40fde [sys.catalog]: configured and running, proceeding with master startup.
I20260926 00:59:54.145284 6972 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a6ef18ce5e544f7a373b7ac86f40fde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a6ef18ce5e544f7a373b7ac86f40fde" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 35839 } }
I20260926 00:59:54.145596 6972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a6ef18ce5e544f7a373b7ac86f40fde [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a6ef18ce5e544f7a373b7ac86f40fde. Latest consensus state: current_term: 1 leader_uuid: "8a6ef18ce5e544f7a373b7ac86f40fde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a6ef18ce5e544f7a373b7ac86f40fde" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 35839 } } }
I20260926 00:59:54.145741 6972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a6ef18ce5e544f7a373b7ac86f40fde [sys.catalog]: This master's current role is: LEADER
I20260926 00:59:54.145630 6974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a6ef18ce5e544f7a373b7ac86f40fde [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a6ef18ce5e544f7a373b7ac86f40fde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a6ef18ce5e544f7a373b7ac86f40fde" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 35839 } } }
I20260926 00:59:54.146538 6974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a6ef18ce5e544f7a373b7ac86f40fde [sys.catalog]: This master's current role is: LEADER
W20260926 00:59:54.147676 6990 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8a6ef18ce5e544f7a373b7ac86f40fde: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 00:59:54.147735 6990 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 00:59:54.147780 6978 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 00:59:54.147917 6978 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 00:59:54.149271 6978 catalog_manager.cc:1347] Generated new cluster ID: db1d8f6252724c5e9344a989fabedd06
I20260926 00:59:54.149322 6978 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 00:59:54 dist-test-slave-7v10 krb5kdc[6888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 00:59:54.156085 6976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:54.145093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51798 (local address 127.3.75.62:35839)
0926 00:59:54.145396 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:54.145426 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:54.145460 (+ 34us) server_negotiation.cc:407] Connection header received
0926 00:59:54.145576 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:54.145583 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:54.145638 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:54.145787 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:54.146527 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:54.147952 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:54.148980 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:54.149584 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:54.149706 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:54.152417 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:54.152421 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:54.152421 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 00:59:54.152433 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:54.154418 (+ 1985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:54.154792 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:54.154795 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:54.154795 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:54.154966 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:54.155168 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:54.155169 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:54.155170 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:54.155491 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:54.155655 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:54.155731 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 00:59:54.155877 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":86,"threads_started":1}
I20260926 00:59:54.163843 6978 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 00:59:54.164405 6978 catalog_manager.cc:1504] Loading token signing keys...
I20260926 00:59:54.170234 6978 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a6ef18ce5e544f7a373b7ac86f40fde: Generated new TSK 0
I20260926 00:59:54.170420 6978 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.75.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab.
Entry for principal kudu/127.3.75.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab.
I20260926 00:59:54.202179 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.1:0
--local_ip_for_outbound_sockets=127.3.75.1
--webserver_interface=127.3.75.1
--webserver_port=0
--tserver_master_addrs=127.3.75.62:35839
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab
--principal=kudu/127.3.75.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:40503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 00:59:54.282456 6995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 00:59:54.282698 6995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 00:59:54.282742 6995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 00:59:54.284006 6995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 00:59:54.284051 6995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 00:59:54.284080 6995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.1
I20260926 00:59:54.285529 6995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:40503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab
--principal=kudu/127.3.75.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.75.1
--webserver_port=0
--tserver_master_addrs=127.3.75.62:35839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.1
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 00:59:54.285718 6995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 00:59:54.285908 6995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 00:59:54.290959 7000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:54.290915 7003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:54.291002 7001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 00:59:54.291201 6995 server_base.cc:1034] running on GCE node
I20260926 00:59:54.291416 6995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 00:59:54.291617 6995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 00:59:54.292773 6995 hybrid_clock.cc:648] HybridClock initialized: now 1790384394292755 us; error 31 us; skew 500 ppm
Sep 26 00:59:54 dist-test-slave-7v10 krb5kdc[6888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384394, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 00:59:54.295361 6995 init.cc:377] Logged in from keytab as kudu/127.3.75.1@KRBTEST.COM (short username kudu)
I20260926 00:59:54.295795 6995 webserver.cc:460] Webserver started at http://127.3.75.1:34643/ using document root <none> and password file <none>
I20260926 00:59:54.296036 6995 fs_manager.cc:359] Metadata directory not provided
I20260926 00:59:54.296079 6995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 00:59:54.296245 6995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 00:59:54.297326 6995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/ts-0/data/instance:
uuid: "5352d449039c48039be1b1b818f9ca3c"
format_stamp: "Formatted at 2026-09-26 00:59:54 on dist-test-slave-7v10"
server_key: "3609e061f609bb13697df2ccfd58920e"
server_key_iv: "142a916d48a5d84ffe3b961ffb9f7c40"
server_key_version: "encryptionkey@0"
I20260926 00:59:54.297689 6995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/ts-0/wal/instance:
uuid: "5352d449039c48039be1b1b818f9ca3c"
format_stamp: "Formatted at 2026-09-26 00:59:54 on dist-test-slave-7v10"
server_key: "3609e061f609bb13697df2ccfd58920e"
server_key_iv: "142a916d48a5d84ffe3b961ffb9f7c40"
server_key_version: "encryptionkey@0"
I20260926 00:59:54.299063 6995 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 00:59:54.299775 7010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:54.299984 6995 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 00:59:54.300056 6995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/ts-0/wal
uuid: "5352d449039c48039be1b1b818f9ca3c"
format_stamp: "Formatted at 2026-09-26 00:59:54 on dist-test-slave-7v10"
server_key: "3609e061f609bb13697df2ccfd58920e"
server_key_iv: "142a916d48a5d84ffe3b961ffb9f7c40"
server_key_version: "encryptionkey@0"
I20260926 00:59:54.300149 6995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 00:59:54.310683 6995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 00:59:54.312446 6995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 00:59:54.312603 6995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 00:59:54.312835 6995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 00:59:54.313115 6995 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 00:59:54.313148 6995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:54.313167 6995 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 00:59:54.313210 6995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:54.318799 6995 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.1:36607
I20260926 00:59:54.318854 7123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.1:36607 every 8 connection(s)
I20260926 00:59:54.319183 6995 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
Sep 26 00:59:54 dist-test-slave-7v10 krb5kdc[6888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384394, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.1@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 00:59:54.326664 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 6995
I20260926 00:59:54.326773 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/ts-0/wal/instance
I20260926 00:59:54.327003 3372 external_mini_cluster.cc:1428] Setting key 1c23ca4bdc2391394357d8e6d772b824
I20260926 00:59:54.328922 6976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:54.319926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:45823 (local address 127.3.75.62:35839)
0926 00:59:54.320007 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:54.320010 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:54.320166 (+ 156us) server_negotiation.cc:407] Connection header received
0926 00:59:54.321096 (+ 930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:54.321097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:54.321108 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:54.321171 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:54.321914 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:54.322430 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:54.323463 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:54.323668 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:54.323734 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:54.326028 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:54.326029 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:54.326030 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 00:59:54.326037 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:54.327494 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:54.327961 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:54.327962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:54.327963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:54.328025 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:54.328272 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:54.328273 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:54.328273 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:54.328394 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:54.328466 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:54.328737 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 00:59:54.328819 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 00:59:54.329263 7126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:54.319886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:35839 (local address 127.3.75.1:45823)
0926 00:59:54.320055 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:54.320088 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:54.320988 (+ 900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:54.321224 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:54.321227 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:54.321571 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0926 00:59:54.321778 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:54.321784 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:54.322516 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:54.322517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:54.323383 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:54.323385 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:54.323763 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:54.323764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:54.323866 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:54.324409 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 00:59:54.324418 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 00:59:54.325944 (+ 1526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 00:59:54.327585 (+ 1641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 00:59:54.327592 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 00:59:54.327595 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 00:59:54.327860 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 00:59:54.328092 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 00:59:54.328093 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 00:59:54.328093 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 00:59:54.328205 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 00:59:54.328470 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 00:59:54.328477 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 00:59:54.328610 (+ 133us) client_negotiation.cc:769] Sending connection context
0926 00:59:54.328715 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 00:59:54.328868 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":98,"threads_started":1}
I20260926 00:59:54.329746 7124 heartbeater.cc:347] Connected to a master server at 127.3.75.62:35839
I20260926 00:59:54.329850 7124 heartbeater.cc:464] Registering TS with master...
I20260926 00:59:54.330040 7124 heartbeater.cc:511] Master 127.3.75.62:35839 requested a full tablet report, sending...
I20260926 00:59:54.330572 6933 ts_manager.cc:194] Registered new tserver with Master: 5352d449039c48039be1b1b818f9ca3c (127.3.75.1:36607)
I20260926 00:59:54.331274 6933 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.75.1@KRBTEST.COM'} at 127.3.75.1:45823
WARNING: no policy specified for kudu/127.3.75.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab.
Entry for principal kudu/127.3.75.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab.
I20260926 00:59:54.351346 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.2:0
--local_ip_for_outbound_sockets=127.3.75.2
--webserver_interface=127.3.75.2
--webserver_port=0
--tserver_master_addrs=127.3.75.62:35839
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab
--principal=kudu/127.3.75.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:40503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 00:59:54.431136 7129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 00:59:54.431336 7129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 00:59:54.431360 7129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 00:59:54.432511 7129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 00:59:54.432554 7129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 00:59:54.432605 7129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.2
I20260926 00:59:54.434150 7129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:40503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab
--principal=kudu/127.3.75.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.75.2
--webserver_port=0
--tserver_master_addrs=127.3.75.62:35839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.2
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 00:59:54.434403 7129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 00:59:54.434638 7129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 00:59:54.439749 7137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:54.439728 7135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:54.439729 7134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 00:59:54.440016 7129 server_base.cc:1034] running on GCE node
I20260926 00:59:54.440215 7129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 00:59:54.440404 7129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 00:59:54.441543 7129 hybrid_clock.cc:648] HybridClock initialized: now 1790384394441525 us; error 39 us; skew 500 ppm
Sep 26 00:59:54 dist-test-slave-7v10 krb5kdc[6888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384394, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 00:59:54.444129 7129 init.cc:377] Logged in from keytab as kudu/127.3.75.2@KRBTEST.COM (short username kudu)
I20260926 00:59:54.444591 7129 webserver.cc:460] Webserver started at http://127.3.75.2:42129/ using document root <none> and password file <none>
I20260926 00:59:54.444816 7129 fs_manager.cc:359] Metadata directory not provided
I20260926 00:59:54.444860 7129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 00:59:54.445008 7129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 00:59:54.446197 7129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/ts-1/data/instance:
uuid: "b8ba54b704034f46b92f54004997c57e"
format_stamp: "Formatted at 2026-09-26 00:59:54 on dist-test-slave-7v10"
server_key: "7060a55d14b0d23ac8656c892d547a17"
server_key_iv: "703f05dd5570e7462ef7aedda7f471d0"
server_key_version: "encryptionkey@0"
I20260926 00:59:54.446587 7129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/ts-1/wal/instance:
uuid: "b8ba54b704034f46b92f54004997c57e"
format_stamp: "Formatted at 2026-09-26 00:59:54 on dist-test-slave-7v10"
server_key: "7060a55d14b0d23ac8656c892d547a17"
server_key_iv: "703f05dd5570e7462ef7aedda7f471d0"
server_key_version: "encryptionkey@0"
I20260926 00:59:54.448107 7129 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 00:59:54.448841 7144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:54.449021 7129 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 00:59:54.449118 7129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/ts-1/wal
uuid: "b8ba54b704034f46b92f54004997c57e"
format_stamp: "Formatted at 2026-09-26 00:59:54 on dist-test-slave-7v10"
server_key: "7060a55d14b0d23ac8656c892d547a17"
server_key_iv: "703f05dd5570e7462ef7aedda7f471d0"
server_key_version: "encryptionkey@0"
I20260926 00:59:54.449203 7129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 00:59:54.459426 7129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 00:59:54.461104 7129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 00:59:54.461256 7129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 00:59:54.461462 7129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 00:59:54.461753 7129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 00:59:54.461807 7129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:54.461845 7129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 00:59:54.461874 7129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:54.467535 7129 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.2:46221
I20260926 00:59:54.467582 7257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.2:46221 every 8 connection(s)
I20260926 00:59:54.467860 7129 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
Sep 26 00:59:54 dist-test-slave-7v10 krb5kdc[6888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384394, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.2@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 00:59:54.475672 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 7129
I20260926 00:59:54.475788 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/ts-1/wal/instance
I20260926 00:59:54.475966 3372 external_mini_cluster.cc:1428] Setting key 5a4a8f773e9af810e24f46a3077e503d
I20260926 00:59:54.476297 6976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:54.468589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:51133 (local address 127.3.75.62:35839)
0926 00:59:54.468636 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:54.468638 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:54.468796 (+ 158us) server_negotiation.cc:407] Connection header received
0926 00:59:54.469714 (+ 918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:54.469715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:54.469725 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:54.469779 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:54.470304 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:54.470770 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:54.471745 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:54.471930 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:54.471981 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:54.473902 (+ 1921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:54.473903 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:54.473904 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 00:59:54.473910 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:54.475043 (+ 1133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:54.475374 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:54.475374 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:54.475374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:54.475421 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:54.475643 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:54.475644 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:54.475645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:54.475779 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:54.475840 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:54.476133 (+ 293us) server_negotiation.cc:299] Negotiation successful
0926 00:59:54.476210 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 00:59:54.476466 7260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:54.468535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:35839 (local address 127.3.75.2:51133)
0926 00:59:54.468692 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:54.468730 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:54.469632 (+ 902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:54.469836 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:54.469839 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:54.470072 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0926 00:59:54.470241 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:54.470244 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:54.470855 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:54.470856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:54.471675 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:54.471677 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:54.471998 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:54.471999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:54.472091 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:54.472501 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 00:59:54.472509 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 00:59:54.473818 (+ 1309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 00:59:54.475106 (+ 1288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 00:59:54.475112 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 00:59:54.475114 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 00:59:54.475306 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 00:59:54.475475 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 00:59:54.475476 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 00:59:54.475476 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 00:59:54.475563 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 00:59:54.475843 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 00:59:54.475850 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 00:59:54.475973 (+ 123us) client_negotiation.cc:769] Sending connection context
0926 00:59:54.476109 (+ 136us) client_negotiation.cc:240] Negotiation successful
0926 00:59:54.476253 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":87,"threads_started":1}
I20260926 00:59:54.476842 7258 heartbeater.cc:347] Connected to a master server at 127.3.75.62:35839
I20260926 00:59:54.476956 7258 heartbeater.cc:464] Registering TS with master...
I20260926 00:59:54.477113 7258 heartbeater.cc:511] Master 127.3.75.62:35839 requested a full tablet report, sending...
I20260926 00:59:54.477463 6933 ts_manager.cc:194] Registered new tserver with Master: b8ba54b704034f46b92f54004997c57e (127.3.75.2:46221)
I20260926 00:59:54.477823 6933 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.75.2@KRBTEST.COM'} at 127.3.75.2:51133
WARNING: no policy specified for kudu/127.3.75.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab.
Entry for principal kudu/127.3.75.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab.
I20260926 00:59:54.498997 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.3:0
--local_ip_for_outbound_sockets=127.3.75.3
--webserver_interface=127.3.75.3
--webserver_port=0
--tserver_master_addrs=127.3.75.62:35839
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab
--principal=kudu/127.3.75.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:40503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 00:59:54.578801 7263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 00:59:54.579092 7263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 00:59:54.579130 7263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 00:59:54.580263 7263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 00:59:54.580331 7263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 00:59:54.580377 7263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.3
I20260926 00:59:54.581864 7263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:40503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab
--principal=kudu/127.3.75.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.75.3
--webserver_port=0
--tserver_master_addrs=127.3.75.62:35839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.3
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 00:59:54.582118 7263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 00:59:54.582337 7263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 00:59:54.587245 7269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:54.587245 7271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 00:59:54.587387 7263 server_base.cc:1034] running on GCE node
W20260926 00:59:54.587286 7268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 00:59:54.587687 7263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 00:59:54.587877 7263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 00:59:54.589047 7263 hybrid_clock.cc:648] HybridClock initialized: now 1790384394589035 us; error 24 us; skew 500 ppm
Sep 26 00:59:54 dist-test-slave-7v10 krb5kdc[6888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384394, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 00:59:54.591419 7263 init.cc:377] Logged in from keytab as kudu/127.3.75.3@KRBTEST.COM (short username kudu)
I20260926 00:59:54.591862 7263 webserver.cc:460] Webserver started at http://127.3.75.3:45191/ using document root <none> and password file <none>
I20260926 00:59:54.592047 7263 fs_manager.cc:359] Metadata directory not provided
I20260926 00:59:54.592085 7263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 00:59:54.592168 7263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 00:59:54.593137 7263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/ts-2/data/instance:
uuid: "c8edb87c3111419c8d05485fa8e13ba9"
format_stamp: "Formatted at 2026-09-26 00:59:54 on dist-test-slave-7v10"
server_key: "e2230fd7f963b6b7d4f8a304f1107a01"
server_key_iv: "d5cff4de6c5f55f4a61d0826f47440fb"
server_key_version: "encryptionkey@0"
I20260926 00:59:54.593423 7263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/ts-2/wal/instance:
uuid: "c8edb87c3111419c8d05485fa8e13ba9"
format_stamp: "Formatted at 2026-09-26 00:59:54 on dist-test-slave-7v10"
server_key: "e2230fd7f963b6b7d4f8a304f1107a01"
server_key_iv: "d5cff4de6c5f55f4a61d0826f47440fb"
server_key_version: "encryptionkey@0"
I20260926 00:59:54.594578 7263 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 00:59:54.595263 7278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:54.595464 7263 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:54.595541 7263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/ts-2/wal
uuid: "c8edb87c3111419c8d05485fa8e13ba9"
format_stamp: "Formatted at 2026-09-26 00:59:54 on dist-test-slave-7v10"
server_key: "e2230fd7f963b6b7d4f8a304f1107a01"
server_key_iv: "d5cff4de6c5f55f4a61d0826f47440fb"
server_key_version: "encryptionkey@0"
I20260926 00:59:54.595613 7263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 00:59:54.614804 7263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 00:59:54.616670 7263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 00:59:54.616827 7263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 00:59:54.617069 7263 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 00:59:54.617379 7263 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 00:59:54.617419 7263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:54.617461 7263 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 00:59:54.617487 7263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:54.623013 7263 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.3:36183
I20260926 00:59:54.623099 7391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.3:36183 every 8 connection(s)
I20260926 00:59:54.623402 7263 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
Sep 26 00:59:54 dist-test-slave-7v10 krb5kdc[6888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384394, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.3@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 00:59:54.631690 6976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:54.624125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.3:33115 (local address 127.3.75.62:35839)
0926 00:59:54.624185 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:54.624187 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:54.624406 (+ 219us) server_negotiation.cc:407] Connection header received
0926 00:59:54.625280 (+ 874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:54.625281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:54.625290 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:54.625340 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:54.625880 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:54.626342 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:54.627257 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:54.627437 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:54.627487 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:54.629409 (+ 1922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:54.629410 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:54.629410 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 00:59:54.629417 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:54.630543 (+ 1126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:54.630938 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:54.630939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:54.630939 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:54.630987 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:54.631194 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:54.631194 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:54.631194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:54.631292 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:54.631342 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:54.631569 (+ 227us) server_negotiation.cc:299] Negotiation successful
0926 00:59:54.631627 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 00:59:54.631873 7394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:54.624121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:35839 (local address 127.3.75.3:33115)
0926 00:59:54.624270 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:54.624303 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:54.625197 (+ 894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:54.625394 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:54.625397 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:54.625640 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0926 00:59:54.625808 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:54.625812 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:54.626415 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:54.626416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:54.627192 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:54.627193 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:54.627504 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:54.627505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:54.627598 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:54.627988 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 00:59:54.627995 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 00:59:54.629325 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 00:59:54.630612 (+ 1287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 00:59:54.630617 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 00:59:54.630618 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 00:59:54.630827 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 00:59:54.631043 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 00:59:54.631044 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 00:59:54.631044 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 00:59:54.631139 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 00:59:54.631345 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 00:59:54.631349 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 00:59:54.631460 (+ 111us) client_negotiation.cc:769] Sending connection context
0926 00:59:54.631542 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 00:59:54.631652 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":87,"threads_started":1}
I20260926 00:59:54.632417 7392 heartbeater.cc:347] Connected to a master server at 127.3.75.62:35839
I20260926 00:59:54.632522 7392 heartbeater.cc:464] Registering TS with master...
I20260926 00:59:54.632699 7392 heartbeater.cc:511] Master 127.3.75.62:35839 requested a full tablet report, sending...
I20260926 00:59:54.633128 6933 ts_manager.cc:194] Registered new tserver with Master: c8edb87c3111419c8d05485fa8e13ba9 (127.3.75.3:36183)
I20260926 00:59:54.633200 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 7263
I20260926 00:59:54.633302 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384372073861-3372-0/minicluster-data/ts-2/wal/instance
I20260926 00:59:54.633507 3372 external_mini_cluster.cc:1428] Setting key c80925fdd3499c9dfed2892edb3a502b
I20260926 00:59:54.633620 6933 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.75.3@KRBTEST.COM'} at 127.3.75.3:33115
I20260926 00:59:54.634390 3372 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 00:59:54 dist-test-slave-7v10 krb5kdc[6888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 00:59:54 dist-test-slave-7v10 krb5kdc[6888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.75.1@KRBTEST.COM
I20260926 00:59:54.657734 7399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:54.649900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58406 (local address 127.3.75.1:36607)
0926 00:59:54.650070 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:54.650079 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:54.650090 (+ 11us) server_negotiation.cc:407] Connection header received
0926 00:59:54.650117 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:54.650119 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:54.650132 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:54.650196 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:54.650726 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:54.651499 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:54.652281 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:54.652599 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:54.652654 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:54.654832 (+ 2178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:54.654842 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:54.654845 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 00:59:54.654858 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:54.656510 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:54.656904 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:54.656908 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:54.656908 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:54.656966 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:54.657163 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:54.657164 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:54.657165 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:54.657319 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:54.657491 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:54.657536 (+ 45us) server_negotiation.cc:299] Negotiation successful
0926 00:59:54.657612 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":100,"threads_started":1}
Sep 26 00:59:54 dist-test-slave-7v10 krb5kdc[6888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 00:59:54.665028 6976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:54.658992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51800 (local address 127.3.75.62:35839)
0926 00:59:54.659066 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:54.659068 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:54.659095 (+ 27us) server_negotiation.cc:407] Connection header received
0926 00:59:54.659148 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:54.659148 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:54.659157 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:54.659201 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:54.659647 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:54.660114 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:54.660725 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:54.660899 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:54.660947 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:54.662985 (+ 2038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:54.662987 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:54.662987 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 00:59:54.662995 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:54.664109 (+ 1114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:54.664396 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:54.664397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:54.664397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:54.664445 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:54.664601 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:54.664602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:54.664602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:54.664732 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:54.664801 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:54.664889 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 00:59:54.664943 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260926 00:59:54.665298 6933 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51800
I20260926 00:59:54.665673 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 6995
I20260926 00:59:54.671380 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 7129
I20260926 00:59:54.675841 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 7263
I20260926 00:59:54.679978 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 6901
2026-09-26T00:59:54Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1809 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 00:59:54 dist-test-slave-7v10 krb5kdc[7404](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 00:59:54 dist-test-slave-7v10 krb5kdc[7404](info): set up 2 sockets
Sep 26 00:59:54 dist-test-slave-7v10 krb5kdc[7404](info): commencing operation
krb5kdc: starting...
W20260926 00:59:55.726006 3372 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Sep 26 00:59:55 dist-test-slave-7v10 krb5kdc[7404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384395, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T00:59:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T00:59:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.75.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab.
Entry for principal kudu/127.3.75.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab.
I20260926 00:59:55.839166 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:33585
--webserver_interface=127.3.75.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:42783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.75.62:33585
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 00:59:55.917760 7417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 00:59:55.917966 7417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 00:59:55.917989 7417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 00:59:55.919103 7417 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 00:59:55.919137 7417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 00:59:55.919152 7417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 00:59:55.919163 7417 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 00:59:55.919175 7417 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 00:59:55.920601 7417 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:42783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.75.62:33585
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:33585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 00:59:55.920836 7417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 00:59:55.921037 7417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 00:59:55.925971 7425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:55.925982 7422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:55.926000 7423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 00:59:55.926271 7417 server_base.cc:1034] running on GCE node
I20260926 00:59:55.926564 7417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 00:59:55.926929 7417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 00:59:55.928100 7417 hybrid_clock.cc:648] HybridClock initialized: now 1790384395928080 us; error 34 us; skew 500 ppm
Sep 26 00:59:55 dist-test-slave-7v10 krb5kdc[7404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384395, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 00:59:55.930523 7417 init.cc:377] Logged in from keytab as kudu/127.3.75.62@KRBTEST.COM (short username kudu)
I20260926 00:59:55.930958 7417 fs_manager.cc:359] Metadata directory not provided
I20260926 00:59:55.931030 7417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 00:59:55.931169 7417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 00:59:55.932235 7417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/master-0/data/instance:
uuid: "d8348538241e48e7b063dff79609ab15"
format_stamp: "Formatted at 2026-09-26 00:59:55 on dist-test-slave-7v10"
server_key: "fa6a0ed61953c8be31f6d7cdac79a105"
server_key_iv: "1c35c7e36063844d6536d377c6dfff81"
server_key_version: "encryptionkey@0"
I20260926 00:59:55.932616 7417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/master-0/wal/instance:
uuid: "d8348538241e48e7b063dff79609ab15"
format_stamp: "Formatted at 2026-09-26 00:59:55 on dist-test-slave-7v10"
server_key: "fa6a0ed61953c8be31f6d7cdac79a105"
server_key_iv: "1c35c7e36063844d6536d377c6dfff81"
server_key_version: "encryptionkey@0"
I20260926 00:59:55.933961 7417 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 00:59:55.934695 7431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:55.934912 7417 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 00:59:55.935022 7417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/master-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/master-0/wal
uuid: "d8348538241e48e7b063dff79609ab15"
format_stamp: "Formatted at 2026-09-26 00:59:55 on dist-test-slave-7v10"
server_key: "fa6a0ed61953c8be31f6d7cdac79a105"
server_key_iv: "1c35c7e36063844d6536d377c6dfff81"
server_key_version: "encryptionkey@0"
I20260926 00:59:55.935114 7417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 00:59:55.942921 7417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 00:59:55.944669 7417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 00:59:55.944828 7417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 00:59:55.948704 7417 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.62:33585
I20260926 00:59:55.948732 7483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.62:33585 every 8 connection(s)
I20260926 00:59:55.949045 7417 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
I20260926 00:59:55.949954 7484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 00:59:55.952245 7484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8348538241e48e7b063dff79609ab15: Bootstrap starting.
I20260926 00:59:55.952832 7484 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8348538241e48e7b063dff79609ab15: Neither blocks nor log segments found. Creating new log.
I20260926 00:59:55.953146 7484 log.cc:824] T 00000000000000000000000000000000 P d8348538241e48e7b063dff79609ab15: Log is configured to *not* fsync() on all Append() calls
I20260926 00:59:55.953210 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 7417
I20260926 00:59:55.953351 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/master-0/wal/instance
I20260926 00:59:55.953583 3372 external_mini_cluster.cc:1428] Setting key d04024fc3379e2941bdcfde786538b2f
I20260926 00:59:55.953930 7484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8348538241e48e7b063dff79609ab15: No bootstrap required, opened a new log
I20260926 00:59:55.955700 7484 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8348538241e48e7b063dff79609ab15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8348538241e48e7b063dff79609ab15" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 33585 } }
I20260926 00:59:55.955840 7484 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8348538241e48e7b063dff79609ab15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 00:59:55.955863 7484 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8348538241e48e7b063dff79609ab15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8348538241e48e7b063dff79609ab15, State: Initialized, Role: FOLLOWER
I20260926 00:59:55.956030 7484 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8348538241e48e7b063dff79609ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8348538241e48e7b063dff79609ab15" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 33585 } }
I20260926 00:59:55.956135 7484 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8348538241e48e7b063dff79609ab15 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 00:59:55.956175 7484 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8348538241e48e7b063dff79609ab15 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 00:59:55.956226 7484 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8348538241e48e7b063dff79609ab15 [term 0 FOLLOWER]: Advancing to term 1
I20260926 00:59:55.957072 7484 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8348538241e48e7b063dff79609ab15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8348538241e48e7b063dff79609ab15" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 33585 } }
I20260926 00:59:55.957190 7484 leader_election.cc:302] T 00000000000000000000000000000000 P d8348538241e48e7b063dff79609ab15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8348538241e48e7b063dff79609ab15; no voters:
I20260926 00:59:55.957369 7484 leader_election.cc:288] T 00000000000000000000000000000000 P d8348538241e48e7b063dff79609ab15 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 00:59:55.957396 7489 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8348538241e48e7b063dff79609ab15 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 00:59:55.957501 7489 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8348538241e48e7b063dff79609ab15 [term 1 LEADER]: Becoming Leader. State: Replica: d8348538241e48e7b063dff79609ab15, State: Running, Role: LEADER
I20260926 00:59:55.957631 7484 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8348538241e48e7b063dff79609ab15 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 00:59:55.957648 7489 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8348538241e48e7b063dff79609ab15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8348538241e48e7b063dff79609ab15" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 33585 } }
I20260926 00:59:55.958832 7491 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8348538241e48e7b063dff79609ab15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8348538241e48e7b063dff79609ab15. Latest consensus state: current_term: 1 leader_uuid: "d8348538241e48e7b063dff79609ab15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8348538241e48e7b063dff79609ab15" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 33585 } } }
I20260926 00:59:55.958987 7491 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8348538241e48e7b063dff79609ab15 [sys.catalog]: This master's current role is: LEADER
I20260926 00:59:55.959270 7490 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8348538241e48e7b063dff79609ab15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8348538241e48e7b063dff79609ab15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8348538241e48e7b063dff79609ab15" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 33585 } } }
I20260926 00:59:55.959358 7490 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8348538241e48e7b063dff79609ab15 [sys.catalog]: This master's current role is: LEADER
W20260926 00:59:55.959590 7504 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d8348538241e48e7b063dff79609ab15: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 00:59:55.959649 7504 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 00:59:55.959761 7505 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 00:59:55.959932 7505 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 00:59:55.961598 7505 catalog_manager.cc:1347] Generated new cluster ID: 305b9d32a00e4ccb99ab84c0fe33684b
I20260926 00:59:55.961653 7505 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 00:59:55 dist-test-slave-7v10 krb5kdc[7404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384395, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 00:59:55.966348 7488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:55.954073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57270 (local address 127.3.75.62:33585)
0926 00:59:55.954266 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:55.954276 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:55.954311 (+ 35us) server_negotiation.cc:407] Connection header received
0926 00:59:55.955689 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:55.955696 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:55.955747 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:55.955909 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:55.956651 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:55.957767 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:55.958962 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:55.959631 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:55.959732 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:55.962611 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:55.962615 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:55.962616 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 00:59:55.962627 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:55.964634 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:55.965020 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:55.965023 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:55.965024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:55.965144 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:55.965350 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:55.965351 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:55.965351 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:55.965720 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:55.965918 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:55.965986 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 00:59:55.966126 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":76,"threads_started":1}
I20260926 00:59:55.983228 7505 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 00:59:55.984009 7505 catalog_manager.cc:1504] Loading token signing keys...
I20260926 00:59:55.990433 7505 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8348538241e48e7b063dff79609ab15: Generated new TSK 0
I20260926 00:59:55.990648 7505 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.75.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab.
Entry for principal kudu/127.3.75.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab.
I20260926 00:59:56.025322 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.1:0
--local_ip_for_outbound_sockets=127.3.75.1
--webserver_interface=127.3.75.1
--webserver_port=0
--tserver_master_addrs=127.3.75.62:33585
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab
--principal=kudu/127.3.75.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:42783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 00:59:56.120515 7510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 00:59:56.120715 7510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 00:59:56.120739 7510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 00:59:56.121943 7510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 00:59:56.121989 7510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 00:59:56.122015 7510 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.1
I20260926 00:59:56.123594 7510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:42783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab
--principal=kudu/127.3.75.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.75.1
--webserver_port=0
--tserver_master_addrs=127.3.75.62:33585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.1
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 00:59:56.123852 7510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 00:59:56.124069 7510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 00:59:56.128986 7518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:56.128996 7516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 00:59:56.129146 7510 server_base.cc:1034] running on GCE node
W20260926 00:59:56.129056 7515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 00:59:56.129436 7510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 00:59:56.129664 7510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 00:59:56.130862 7510 hybrid_clock.cc:648] HybridClock initialized: now 1790384396130840 us; error 37 us; skew 500 ppm
Sep 26 00:59:56 dist-test-slave-7v10 krb5kdc[7404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384396, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 00:59:56.133388 7510 init.cc:377] Logged in from keytab as kudu/127.3.75.1@KRBTEST.COM (short username kudu)
I20260926 00:59:56.133764 7510 fs_manager.cc:359] Metadata directory not provided
I20260926 00:59:56.133836 7510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 00:59:56.133978 7510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 00:59:56.135089 7510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/ts-0/data/instance:
uuid: "6d8b211f4cdd433abc56965aa8d0b801"
format_stamp: "Formatted at 2026-09-26 00:59:56 on dist-test-slave-7v10"
server_key: "ec947889d58b5485a199f11e4e399e26"
server_key_iv: "1a62a62c0964dbc333452e0e59b28331"
server_key_version: "encryptionkey@0"
I20260926 00:59:56.135447 7510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/ts-0/wal/instance:
uuid: "6d8b211f4cdd433abc56965aa8d0b801"
format_stamp: "Formatted at 2026-09-26 00:59:56 on dist-test-slave-7v10"
server_key: "ec947889d58b5485a199f11e4e399e26"
server_key_iv: "1a62a62c0964dbc333452e0e59b28331"
server_key_version: "encryptionkey@0"
I20260926 00:59:56.136698 7510 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 00:59:56.137341 7524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:56.137682 7510 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 00:59:56.137781 7510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/ts-0/wal
uuid: "6d8b211f4cdd433abc56965aa8d0b801"
format_stamp: "Formatted at 2026-09-26 00:59:56 on dist-test-slave-7v10"
server_key: "ec947889d58b5485a199f11e4e399e26"
server_key_iv: "1a62a62c0964dbc333452e0e59b28331"
server_key_version: "encryptionkey@0"
I20260926 00:59:56.137876 7510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 00:59:56.154549 7510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 00:59:56.156898 7510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 00:59:56.157058 7510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 00:59:56.157284 7510 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 00:59:56.157660 7510 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 00:59:56.157723 7510 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:56.157766 7510 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 00:59:56.157793 7510 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:56.163779 7510 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.1:37333
I20260926 00:59:56.163825 7637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.1:37333 every 8 connection(s)
I20260926 00:59:56.164095 7510 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
Sep 26 00:59:56 dist-test-slave-7v10 krb5kdc[7404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384396, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.1@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 00:59:56.171021 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 7510
I20260926 00:59:56.171144 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/ts-0/wal/instance
I20260926 00:59:56.171327 3372 external_mini_cluster.cc:1428] Setting key c6be52a3ffa17eaf8bb3db346413b40c
I20260926 00:59:56.174144 7640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:56.164802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:33585 (local address 127.3.75.1:41167)
0926 00:59:56.164991 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:56.165027 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:56.165522 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:56.165782 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:56.165785 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:56.166052 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0926 00:59:56.166547 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:56.166552 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:56.167346 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:56.167347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:56.168218 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:56.168220 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:56.168640 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:56.168641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:56.168764 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:56.169307 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 00:59:56.169315 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 00:59:56.170721 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 00:59:56.172649 (+ 1928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 00:59:56.172657 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 00:59:56.172659 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 00:59:56.173006 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 00:59:56.173195 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 00:59:56.173195 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 00:59:56.173196 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 00:59:56.173346 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 00:59:56.173649 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 00:59:56.173654 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 00:59:56.173771 (+ 117us) client_negotiation.cc:769] Sending connection context
0926 00:59:56.173862 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 00:59:56.173984 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":90,"threads_started":1}
I20260926 00:59:56.174139 7488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:56.164842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:41167 (local address 127.3.75.62:33585)
0926 00:59:56.164928 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:56.164931 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:56.165113 (+ 182us) server_negotiation.cc:407] Connection header received
0926 00:59:56.165630 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:56.165631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:56.165648 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:56.165699 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:56.166651 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:56.167250 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:56.168317 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:56.168539 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:56.168628 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:56.170828 (+ 2200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:56.170830 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:56.170830 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 00:59:56.170839 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:56.172567 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:56.173081 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:56.173081 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:56.173082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:56.173133 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:56.173431 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:56.173432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:56.173433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:56.173579 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:56.173652 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:56.173907 (+ 255us) server_negotiation.cc:299] Negotiation successful
0926 00:59:56.174020 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 00:59:56.174623 7638 heartbeater.cc:347] Connected to a master server at 127.3.75.62:33585
I20260926 00:59:56.174734 7638 heartbeater.cc:464] Registering TS with master...
I20260926 00:59:56.174944 7638 heartbeater.cc:511] Master 127.3.75.62:33585 requested a full tablet report, sending...
I20260926 00:59:56.175491 7448 ts_manager.cc:194] Registered new tserver with Master: 6d8b211f4cdd433abc56965aa8d0b801 (127.3.75.1:37333)
I20260926 00:59:56.176180 7448 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.75.1@KRBTEST.COM'} at 127.3.75.1:41167
WARNING: no policy specified for kudu/127.3.75.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab.
Entry for principal kudu/127.3.75.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab.
I20260926 00:59:56.197427 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.2:0
--local_ip_for_outbound_sockets=127.3.75.2
--webserver_interface=127.3.75.2
--webserver_port=0
--tserver_master_addrs=127.3.75.62:33585
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab
--principal=kudu/127.3.75.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:42783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 00:59:56.276316 7643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 00:59:56.276552 7643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 00:59:56.276587 7643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 00:59:56.277671 7643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 00:59:56.277738 7643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 00:59:56.277781 7643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.2
I20260926 00:59:56.279240 7643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:42783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab
--principal=kudu/127.3.75.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.75.2
--webserver_port=0
--tserver_master_addrs=127.3.75.62:33585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.2
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 00:59:56.279510 7643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 00:59:56.279731 7643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 00:59:56.284549 7643 server_base.cc:1034] running on GCE node
W20260926 00:59:56.284616 7651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:56.284562 7648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:56.284584 7649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 00:59:56.285051 7643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 00:59:56.285256 7643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 00:59:56.286391 7643 hybrid_clock.cc:648] HybridClock initialized: now 1790384396286381 us; error 28 us; skew 500 ppm
Sep 26 00:59:56 dist-test-slave-7v10 krb5kdc[7404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384396, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 00:59:56.288875 7643 init.cc:377] Logged in from keytab as kudu/127.3.75.2@KRBTEST.COM (short username kudu)
I20260926 00:59:56.289253 7643 fs_manager.cc:359] Metadata directory not provided
I20260926 00:59:56.289324 7643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 00:59:56.289458 7643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 00:59:56.290458 7643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/ts-1/data/instance:
uuid: "53c5c6ea4be4420c93ba845b01cf4420"
format_stamp: "Formatted at 2026-09-26 00:59:56 on dist-test-slave-7v10"
server_key: "26e513d5141bdc3fd39b3812165d16d3"
server_key_iv: "0c13da53b4260963879d6999e5fb5fee"
server_key_version: "encryptionkey@0"
I20260926 00:59:56.290813 7643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/ts-1/wal/instance:
uuid: "53c5c6ea4be4420c93ba845b01cf4420"
format_stamp: "Formatted at 2026-09-26 00:59:56 on dist-test-slave-7v10"
server_key: "26e513d5141bdc3fd39b3812165d16d3"
server_key_iv: "0c13da53b4260963879d6999e5fb5fee"
server_key_version: "encryptionkey@0"
I20260926 00:59:56.292105 7643 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 00:59:56.292805 7657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:56.292979 7643 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 00:59:56.293076 7643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/ts-1/wal
uuid: "53c5c6ea4be4420c93ba845b01cf4420"
format_stamp: "Formatted at 2026-09-26 00:59:56 on dist-test-slave-7v10"
server_key: "26e513d5141bdc3fd39b3812165d16d3"
server_key_iv: "0c13da53b4260963879d6999e5fb5fee"
server_key_version: "encryptionkey@0"
I20260926 00:59:56.293162 7643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 00:59:56.315853 7643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 00:59:56.317628 7643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 00:59:56.317790 7643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 00:59:56.318014 7643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 00:59:56.318342 7643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 00:59:56.318399 7643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:56.318434 7643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 00:59:56.318463 7643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:56.324440 7643 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.2:45281
I20260926 00:59:56.324486 7770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.2:45281 every 8 connection(s)
I20260926 00:59:56.324757 7643 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
Sep 26 00:59:56 dist-test-slave-7v10 krb5kdc[7404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384396, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.2@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 00:59:56.332260 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 7643
I20260926 00:59:56.332379 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/ts-1/wal/instance
I20260926 00:59:56.332600 3372 external_mini_cluster.cc:1428] Setting key 0ccf39ff3e31f615f9b112383c773cf9
I20260926 00:59:56.334093 7488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:56.325493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:42845 (local address 127.3.75.62:33585)
0926 00:59:56.325569 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:56.325572 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:56.325723 (+ 151us) server_negotiation.cc:407] Connection header received
0926 00:59:56.326184 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:56.326185 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:56.326196 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:56.326254 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:56.327228 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:56.327725 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:56.328663 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:56.328840 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:56.328888 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:56.331025 (+ 2137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:56.331026 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:56.331027 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 00:59:56.331032 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:56.332660 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:56.333100 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:56.333101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:56.333102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:56.333164 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:56.333460 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:56.333461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:56.333461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:56.333576 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:56.333651 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:56.333918 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 00:59:56.334004 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 00:59:56.334275 7773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:56.325406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:33585 (local address 127.3.75.2:42845)
0926 00:59:56.325590 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:56.325631 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:56.326099 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:56.326307 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:56.326310 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:56.326572 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0926 00:59:56.327143 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:56.327150 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:56.327802 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:56.327803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:56.328591 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:56.328593 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:56.328906 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:56.328907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:56.328999 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:56.329527 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 00:59:56.329535 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 00:59:56.330940 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 00:59:56.332759 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 00:59:56.332765 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 00:59:56.332767 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 00:59:56.333037 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 00:59:56.333259 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 00:59:56.333259 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 00:59:56.333260 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 00:59:56.333380 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 00:59:56.333644 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 00:59:56.333650 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 00:59:56.333786 (+ 136us) client_negotiation.cc:769] Sending connection context
0926 00:59:56.333897 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 00:59:56.334049 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":91,"threads_started":1}
I20260926 00:59:56.334946 7771 heartbeater.cc:347] Connected to a master server at 127.3.75.62:33585
I20260926 00:59:56.335055 7771 heartbeater.cc:464] Registering TS with master...
I20260926 00:59:56.335212 7771 heartbeater.cc:511] Master 127.3.75.62:33585 requested a full tablet report, sending...
I20260926 00:59:56.335639 7448 ts_manager.cc:194] Registered new tserver with Master: 53c5c6ea4be4420c93ba845b01cf4420 (127.3.75.2:45281)
I20260926 00:59:56.336143 7448 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.75.2@KRBTEST.COM'} at 127.3.75.2:42845
WARNING: no policy specified for kudu/127.3.75.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab.
Entry for principal kudu/127.3.75.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab.
I20260926 00:59:56.356935 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.3:0
--local_ip_for_outbound_sockets=127.3.75.3
--webserver_interface=127.3.75.3
--webserver_port=0
--tserver_master_addrs=127.3.75.62:33585
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab
--principal=kudu/127.3.75.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:42783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 00:59:56.436409 7776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 00:59:56.436596 7776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 00:59:56.436619 7776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 00:59:56.437800 7776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 00:59:56.437841 7776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 00:59:56.437870 7776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.3
I20260926 00:59:56.439450 7776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:42783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab
--principal=kudu/127.3.75.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.75.3
--webserver_port=0
--tserver_master_addrs=127.3.75.62:33585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.3
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 00:59:56.439668 7776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 00:59:56.439903 7776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 00:59:56.445024 7784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:56.445017 7781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:56.445017 7782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 00:59:56.445226 7776 server_base.cc:1034] running on GCE node
I20260926 00:59:56.445487 7776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 00:59:56.445726 7776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 00:59:56.446914 7776 hybrid_clock.cc:648] HybridClock initialized: now 1790384396446856 us; error 29 us; skew 500 ppm
Sep 26 00:59:56 dist-test-slave-7v10 krb5kdc[7404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384396, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 00:59:56.449313 7776 init.cc:377] Logged in from keytab as kudu/127.3.75.3@KRBTEST.COM (short username kudu)
I20260926 00:59:56.449644 7776 fs_manager.cc:359] Metadata directory not provided
I20260926 00:59:56.449689 7776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 00:59:56.449856 7776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 00:59:56.450933 7776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/ts-2/data/instance:
uuid: "739b62f047fb4a3ba140ea7fc64d4112"
format_stamp: "Formatted at 2026-09-26 00:59:56 on dist-test-slave-7v10"
server_key: "beebf959567b8a38aa0fd31c43c17a51"
server_key_iv: "a7f0448257974a224e59aad37bf5b406"
server_key_version: "encryptionkey@0"
I20260926 00:59:56.451298 7776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/ts-2/wal/instance:
uuid: "739b62f047fb4a3ba140ea7fc64d4112"
format_stamp: "Formatted at 2026-09-26 00:59:56 on dist-test-slave-7v10"
server_key: "beebf959567b8a38aa0fd31c43c17a51"
server_key_iv: "a7f0448257974a224e59aad37bf5b406"
server_key_version: "encryptionkey@0"
I20260926 00:59:56.452527 7776 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 00:59:56.453186 7790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:56.453384 7776 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 00:59:56.453454 7776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/ts-2/wal
uuid: "739b62f047fb4a3ba140ea7fc64d4112"
format_stamp: "Formatted at 2026-09-26 00:59:56 on dist-test-slave-7v10"
server_key: "beebf959567b8a38aa0fd31c43c17a51"
server_key_iv: "a7f0448257974a224e59aad37bf5b406"
server_key_version: "encryptionkey@0"
I20260926 00:59:56.453552 7776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 00:59:56.466720 7776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 00:59:56.468443 7776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 00:59:56.468590 7776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 00:59:56.468801 7776 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 00:59:56.469102 7776 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 00:59:56.469156 7776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:56.469197 7776 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 00:59:56.469225 7776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:56.474710 7776 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.3:34953
I20260926 00:59:56.474781 7903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.3:34953 every 8 connection(s)
I20260926 00:59:56.475069 7776 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
Sep 26 00:59:56 dist-test-slave-7v10 krb5kdc[7404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384396, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.3@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 00:59:56.481899 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 7776
I20260926 00:59:56.482025 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/ts-2/wal/instance
I20260926 00:59:56.482216 3372 external_mini_cluster.cc:1428] Setting key 94c1d3737c51a0128025f93669eb507b
I20260926 00:59:56.484144 7488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:56.475791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.3:52575 (local address 127.3.75.62:33585)
0926 00:59:56.475858 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:56.475860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:56.476055 (+ 195us) server_negotiation.cc:407] Connection header received
0926 00:59:56.476537 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:56.476537 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:56.476548 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:56.476608 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:56.477467 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:56.477962 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:56.478923 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:56.479115 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:56.479167 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:56.481079 (+ 1912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:56.481081 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:56.481081 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 00:59:56.481089 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:56.482712 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:56.483159 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:56.483161 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:56.483161 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:56.483239 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:56.483515 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:56.483516 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:56.483516 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:56.483631 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:56.483701 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:56.483957 (+ 256us) server_negotiation.cc:299] Negotiation successful
0926 00:59:56.484058 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 00:59:56.484333 7906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:56.475775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:33585 (local address 127.3.75.3:52575)
0926 00:59:56.475934 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:56.475973 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:56.476455 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:56.476661 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:56.476665 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:56.476910 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0926 00:59:56.477391 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:56.477396 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:56.478057 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:56.478058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:56.478827 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:56.478829 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:56.479185 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:56.479185 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:56.479278 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:56.479679 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 00:59:56.479687 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 00:59:56.480986 (+ 1299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 00:59:56.482802 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 00:59:56.482808 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 00:59:56.482810 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 00:59:56.483080 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 00:59:56.483323 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 00:59:56.483324 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 00:59:56.483324 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 00:59:56.483427 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 00:59:56.483702 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 00:59:56.483708 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 00:59:56.483827 (+ 119us) client_negotiation.cc:769] Sending connection context
0926 00:59:56.483933 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 00:59:56.484099 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":95,"threads_started":1}
I20260926 00:59:56.484732 7904 heartbeater.cc:347] Connected to a master server at 127.3.75.62:33585
I20260926 00:59:56.484812 7904 heartbeater.cc:464] Registering TS with master...
I20260926 00:59:56.484987 7904 heartbeater.cc:511] Master 127.3.75.62:33585 requested a full tablet report, sending...
I20260926 00:59:56.485357 7448 ts_manager.cc:194] Registered new tserver with Master: 739b62f047fb4a3ba140ea7fc64d4112 (127.3.75.3:34953)
I20260926 00:59:56.485728 7448 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.75.3@KRBTEST.COM'} at 127.3.75.3:52575
I20260926 00:59:56.493883 3372 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 00:59:56.500613 7488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:56.495064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57278 (local address 127.3.75.62:33585)
0926 00:59:56.495136 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:56.495138 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:56.495220 (+ 82us) server_negotiation.cc:407] Connection header received
0926 00:59:56.495280 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:56.495280 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:56.495290 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:56.495338 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:56.495878 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:56.496297 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:56.496970 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:56.497157 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:56.497235 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:56.498320 (+ 1085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:56.498322 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:56.498322 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 00:59:56.498328 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:56.499480 (+ 1152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:56.499830 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:56.499831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:56.499831 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:56.499888 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:56.500096 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:56.500097 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:56.500097 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:56.500234 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:56.500331 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:56.500451 (+ 120us) server_negotiation.cc:299] Negotiation successful
0926 00:59:56.500521 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 00:59:56.501976 7448 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57278:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 00:59:56.503484 7448 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 00:59:56.510054 7917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:56.505590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.1:37333 (local address 127.0.0.1:39090)
0926 00:59:56.506128 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:56.506136 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:56.506180 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:56.506354 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:56.506357 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:56.506360 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 00:59:56.506421 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:56.506422 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:56.507746 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:56.507747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:56.508883 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:56.508886 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:56.509706 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:56.509707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:56.509792 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:56.509805 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 00:59:56.509887 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 00:59:56.509946 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"mutex_wait_us":56,"thread_start_us":80,"threads_started":1}
I20260926 00:59:56.510051 7919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:56.505823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39090 (local address 127.3.75.1:37333)
0926 00:59:56.505985 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:56.505989 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:56.506174 (+ 185us) server_negotiation.cc:407] Connection header received
0926 00:59:56.506230 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:56.506232 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:56.506244 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:56.506341 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 00:59:56.506481 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:56.507636 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:56.508971 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:56.509614 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:56.509689 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:56.509742 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:56.509917 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 00:59:56.509963 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":87,"threads_started":1}
I20260926 00:59:56.510170 7918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:56.505631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50818 (local address 127.3.75.3:34953)
0926 00:59:56.506482 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:56.506487 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:56.506496 (+ 9us) server_negotiation.cc:407] Connection header received
0926 00:59:56.506543 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:56.506544 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:56.506559 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:56.506805 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 00:59:56.506906 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:56.507618 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:56.509162 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:56.509685 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:56.509750 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:56.509806 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:56.510032 (+ 226us) server_negotiation.cc:299] Negotiation successful
0926 00:59:56.510077 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":816,"thread_start_us":67,"threads_started":1}
I20260926 00:59:56.510356 7920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:56.506009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49194 (local address 127.3.75.2:45281)
0926 00:59:56.506123 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:56.506126 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:56.506314 (+ 188us) server_negotiation.cc:407] Connection header received
0926 00:59:56.506595 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:56.506598 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:56.506612 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:56.506690 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 00:59:56.506807 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:56.507977 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:56.509239 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:56.509945 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:56.510010 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:56.510061 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:56.510240 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 00:59:56.510278 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":62,"threads_started":1}
I20260926 00:59:56.510332 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:56.505590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.2:45281 (local address 127.0.0.1:49194)
0926 00:59:56.506236 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:56.506242 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:56.506360 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:56.506687 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:56.506688 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:56.506690 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 00:59:56.506731 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:56.506732 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:56.508074 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:56.508075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:56.509162 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:56.509164 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:56.510099 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:56.510100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:56.510168 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:56.510172 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 00:59:56.510226 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 00:59:56.510264 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"mutex_wait_us":2,"thread_start_us":73,"threads_started":1}
I20260926 00:59:56.510556 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:56.505561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.3:34953 (local address 127.0.0.1:50818)
0926 00:59:56.505712 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:56.505747 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:56.505813 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:56.506768 (+ 955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:56.506769 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:56.506770 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 00:59:56.506809 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:56.506810 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:56.507723 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:56.507724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:56.509066 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:56.509069 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:56.509838 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:56.509839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:56.509898 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:56.509900 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 00:59:56.509946 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 00:59:56.509974 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":97,"thread_start_us":55,"threads_started":1}
I20260926 00:59:56.511384 7572 tablet_service.cc:1467] Processing CreateTablet for tablet 247bddda55104f219ff5a26b18002466 (DEFAULT_TABLE table=test-table [id=691136cc760147c58ebc4131e5330790]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 00:59:56.511430 7705 tablet_service.cc:1467] Processing CreateTablet for tablet 247bddda55104f219ff5a26b18002466 (DEFAULT_TABLE table=test-table [id=691136cc760147c58ebc4131e5330790]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 00:59:56.511428 7838 tablet_service.cc:1467] Processing CreateTablet for tablet 247bddda55104f219ff5a26b18002466 (DEFAULT_TABLE table=test-table [id=691136cc760147c58ebc4131e5330790]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 00:59:56.511735 7572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 247bddda55104f219ff5a26b18002466. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 00:59:56.511740 7705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 247bddda55104f219ff5a26b18002466. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 00:59:56.511740 7838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 247bddda55104f219ff5a26b18002466. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 00:59:56.514253 7921 tablet_bootstrap.cc:492] T 247bddda55104f219ff5a26b18002466 P 6d8b211f4cdd433abc56965aa8d0b801: Bootstrap starting.
I20260926 00:59:56.514959 7921 tablet_bootstrap.cc:654] T 247bddda55104f219ff5a26b18002466 P 6d8b211f4cdd433abc56965aa8d0b801: Neither blocks nor log segments found. Creating new log.
I20260926 00:59:56.515175 7923 tablet_bootstrap.cc:492] T 247bddda55104f219ff5a26b18002466 P 53c5c6ea4be4420c93ba845b01cf4420: Bootstrap starting.
I20260926 00:59:56.515677 7921 log.cc:824] T 247bddda55104f219ff5a26b18002466 P 6d8b211f4cdd433abc56965aa8d0b801: Log is configured to *not* fsync() on all Append() calls
I20260926 00:59:56.515839 7923 tablet_bootstrap.cc:654] T 247bddda55104f219ff5a26b18002466 P 53c5c6ea4be4420c93ba845b01cf4420: Neither blocks nor log segments found. Creating new log.
I20260926 00:59:56.516422 7923 log.cc:824] T 247bddda55104f219ff5a26b18002466 P 53c5c6ea4be4420c93ba845b01cf4420: Log is configured to *not* fsync() on all Append() calls
I20260926 00:59:56.516530 7921 tablet_bootstrap.cc:492] T 247bddda55104f219ff5a26b18002466 P 6d8b211f4cdd433abc56965aa8d0b801: No bootstrap required, opened a new log
I20260926 00:59:56.516597 7921 ts_tablet_manager.cc:1397] T 247bddda55104f219ff5a26b18002466 P 6d8b211f4cdd433abc56965aa8d0b801: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 00:59:56.516726 7922 tablet_bootstrap.cc:492] T 247bddda55104f219ff5a26b18002466 P 739b62f047fb4a3ba140ea7fc64d4112: Bootstrap starting.
I20260926 00:59:56.517117 7923 tablet_bootstrap.cc:492] T 247bddda55104f219ff5a26b18002466 P 53c5c6ea4be4420c93ba845b01cf4420: No bootstrap required, opened a new log
I20260926 00:59:56.517195 7923 ts_tablet_manager.cc:1397] T 247bddda55104f219ff5a26b18002466 P 53c5c6ea4be4420c93ba845b01cf4420: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 00:59:56.517338 7922 tablet_bootstrap.cc:654] T 247bddda55104f219ff5a26b18002466 P 739b62f047fb4a3ba140ea7fc64d4112: Neither blocks nor log segments found. Creating new log.
I20260926 00:59:56.517903 7922 log.cc:824] T 247bddda55104f219ff5a26b18002466 P 739b62f047fb4a3ba140ea7fc64d4112: Log is configured to *not* fsync() on all Append() calls
I20260926 00:59:56.518707 7922 tablet_bootstrap.cc:492] T 247bddda55104f219ff5a26b18002466 P 739b62f047fb4a3ba140ea7fc64d4112: No bootstrap required, opened a new log
I20260926 00:59:56.518777 7922 ts_tablet_manager.cc:1397] T 247bddda55104f219ff5a26b18002466 P 739b62f047fb4a3ba140ea7fc64d4112: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260926 00:59:56.518760 7921 raft_consensus.cc:348] T 247bddda55104f219ff5a26b18002466 P 6d8b211f4cdd433abc56965aa8d0b801 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "739b62f047fb4a3ba140ea7fc64d4112" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 34953 } } peers { permanent_uuid: "6d8b211f4cdd433abc56965aa8d0b801" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37333 } } peers { permanent_uuid: "53c5c6ea4be4420c93ba845b01cf4420" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 45281 } }
I20260926 00:59:56.518957 7921 raft_consensus.cc:374] T 247bddda55104f219ff5a26b18002466 P 6d8b211f4cdd433abc56965aa8d0b801 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 00:59:56.519001 7921 raft_consensus.cc:729] T 247bddda55104f219ff5a26b18002466 P 6d8b211f4cdd433abc56965aa8d0b801 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d8b211f4cdd433abc56965aa8d0b801, State: Initialized, Role: FOLLOWER
I20260926 00:59:56.519121 7921 consensus_queue.cc:260] T 247bddda55104f219ff5a26b18002466 P 6d8b211f4cdd433abc56965aa8d0b801 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "739b62f047fb4a3ba140ea7fc64d4112" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 34953 } } peers { permanent_uuid: "6d8b211f4cdd433abc56965aa8d0b801" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37333 } } peers { permanent_uuid: "53c5c6ea4be4420c93ba845b01cf4420" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 45281 } }
I20260926 00:59:56.519079 7923 raft_consensus.cc:348] T 247bddda55104f219ff5a26b18002466 P 53c5c6ea4be4420c93ba845b01cf4420 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "739b62f047fb4a3ba140ea7fc64d4112" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 34953 } } peers { permanent_uuid: "6d8b211f4cdd433abc56965aa8d0b801" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37333 } } peers { permanent_uuid: "53c5c6ea4be4420c93ba845b01cf4420" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 45281 } }
I20260926 00:59:56.519266 7923 raft_consensus.cc:374] T 247bddda55104f219ff5a26b18002466 P 53c5c6ea4be4420c93ba845b01cf4420 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 00:59:56.519305 7923 raft_consensus.cc:729] T 247bddda55104f219ff5a26b18002466 P 53c5c6ea4be4420c93ba845b01cf4420 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53c5c6ea4be4420c93ba845b01cf4420, State: Initialized, Role: FOLLOWER
I20260926 00:59:56.519393 7921 ts_tablet_manager.cc:1428] T 247bddda55104f219ff5a26b18002466 P 6d8b211f4cdd433abc56965aa8d0b801: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260926 00:59:56.519428 7923 consensus_queue.cc:260] T 247bddda55104f219ff5a26b18002466 P 53c5c6ea4be4420c93ba845b01cf4420 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "739b62f047fb4a3ba140ea7fc64d4112" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 34953 } } peers { permanent_uuid: "6d8b211f4cdd433abc56965aa8d0b801" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37333 } } peers { permanent_uuid: "53c5c6ea4be4420c93ba845b01cf4420" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 45281 } }
I20260926 00:59:56.519521 7638 heartbeater.cc:503] Master 127.3.75.62:33585 was elected leader, sending a full tablet report...
I20260926 00:59:56.519712 7923 ts_tablet_manager.cc:1428] T 247bddda55104f219ff5a26b18002466 P 53c5c6ea4be4420c93ba845b01cf4420: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260926 00:59:56.519739 7771 heartbeater.cc:503] Master 127.3.75.62:33585 was elected leader, sending a full tablet report...
I20260926 00:59:56.520933 7922 raft_consensus.cc:348] T 247bddda55104f219ff5a26b18002466 P 739b62f047fb4a3ba140ea7fc64d4112 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "739b62f047fb4a3ba140ea7fc64d4112" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 34953 } } peers { permanent_uuid: "6d8b211f4cdd433abc56965aa8d0b801" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37333 } } peers { permanent_uuid: "53c5c6ea4be4420c93ba845b01cf4420" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 45281 } }
I20260926 00:59:56.521062 7922 raft_consensus.cc:374] T 247bddda55104f219ff5a26b18002466 P 739b62f047fb4a3ba140ea7fc64d4112 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 00:59:56.521082 7922 raft_consensus.cc:729] T 247bddda55104f219ff5a26b18002466 P 739b62f047fb4a3ba140ea7fc64d4112 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 739b62f047fb4a3ba140ea7fc64d4112, State: Initialized, Role: FOLLOWER
I20260926 00:59:56.521209 7922 consensus_queue.cc:260] T 247bddda55104f219ff5a26b18002466 P 739b62f047fb4a3ba140ea7fc64d4112 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "739b62f047fb4a3ba140ea7fc64d4112" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 34953 } } peers { permanent_uuid: "6d8b211f4cdd433abc56965aa8d0b801" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37333 } } peers { permanent_uuid: "53c5c6ea4be4420c93ba845b01cf4420" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 45281 } }
I20260926 00:59:56.521497 7922 ts_tablet_manager.cc:1428] T 247bddda55104f219ff5a26b18002466 P 739b62f047fb4a3ba140ea7fc64d4112: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260926 00:59:56.521492 7904 heartbeater.cc:503] Master 127.3.75.62:33585 was elected leader, sending a full tablet report...
W20260926 00:59:56.575373 7772 tablet.cc:2369] T 247bddda55104f219ff5a26b18002466 P 53c5c6ea4be4420c93ba845b01cf4420: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 00:59:56.596727 7928 raft_consensus.cc:482] T 247bddda55104f219ff5a26b18002466 P 53c5c6ea4be4420c93ba845b01cf4420 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 00:59:56.596840 7928 raft_consensus.cc:504] T 247bddda55104f219ff5a26b18002466 P 53c5c6ea4be4420c93ba845b01cf4420 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "739b62f047fb4a3ba140ea7fc64d4112" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 34953 } } peers { permanent_uuid: "6d8b211f4cdd433abc56965aa8d0b801" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37333 } } peers { permanent_uuid: "53c5c6ea4be4420c93ba845b01cf4420" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 45281 } }
I20260926 00:59:56.597139 7928 leader_election.cc:288] T 247bddda55104f219ff5a26b18002466 P 53c5c6ea4be4420c93ba845b01cf4420 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 739b62f047fb4a3ba140ea7fc64d4112 (127.3.75.3:34953), 6d8b211f4cdd433abc56965aa8d0b801 (127.3.75.1:37333)
I20260926 00:59:56.601037 7919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:56.597422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:46441 (local address 127.3.75.1:37333)
0926 00:59:56.597465 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:56.597467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:56.597478 (+ 11us) server_negotiation.cc:407] Connection header received
0926 00:59:56.597508 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:56.597509 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:56.597520 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:56.597604 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 00:59:56.597790 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:56.598543 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:56.600080 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:56.600640 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:56.600719 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:56.600732 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:56.600916 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 00:59:56.600968 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 00:59:56.601037 7918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:56.597409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:47953 (local address 127.3.75.3:34953)
0926 00:59:56.597481 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:56.597484 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:56.597630 (+ 146us) server_negotiation.cc:407] Connection header received
0926 00:59:56.597656 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:56.597657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:56.597674 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:56.597713 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 00:59:56.597867 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:56.598553 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:56.600079 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:56.600674 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:56.600738 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:56.600748 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:56.600916 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 00:59:56.600968 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 00:59:56.601078 7773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:56.597338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.1:37333 (local address 127.3.75.2:46441)
0926 00:59:56.597394 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:56.597407 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:56.597449 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:56.597608 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:56.597609 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:56.597611 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 00:59:56.597687 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:56.597689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:56.598638 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:56.598639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:56.599987 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:56.599990 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:56.600733 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:56.600734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:56.600816 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:56.600820 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 00:59:56.600886 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 00:59:56.600929 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":77}
I20260926 00:59:56.601087 7930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:56.597347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.3:34953 (local address 127.3.75.2:47953)
0926 00:59:56.597532 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:56.597540 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:56.597595 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:56.597735 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:56.597736 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:56.597737 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 00:59:56.597797 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:56.597798 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:56.598638 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:56.598639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:56.599987 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:56.599990 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:56.600748 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:56.600748 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:56.600818 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:56.600820 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 00:59:56.600881 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 00:59:56.600928 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":53,"thread_start_us":113,"threads_started":1}
I20260926 00:59:56.601490 7592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "247bddda55104f219ff5a26b18002466" candidate_uuid: "53c5c6ea4be4420c93ba845b01cf4420" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d8b211f4cdd433abc56965aa8d0b801" is_pre_election: true
I20260926 00:59:56.601540 7858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "247bddda55104f219ff5a26b18002466" candidate_uuid: "53c5c6ea4be4420c93ba845b01cf4420" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "739b62f047fb4a3ba140ea7fc64d4112" is_pre_election: true
I20260926 00:59:56.601649 7858 raft_consensus.cc:2454] T 247bddda55104f219ff5a26b18002466 P 739b62f047fb4a3ba140ea7fc64d4112 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53c5c6ea4be4420c93ba845b01cf4420 in term 0.
I20260926 00:59:56.601649 7592 raft_consensus.cc:2454] T 247bddda55104f219ff5a26b18002466 P 6d8b211f4cdd433abc56965aa8d0b801 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53c5c6ea4be4420c93ba845b01cf4420 in term 0.
I20260926 00:59:56.601859 7658 leader_election.cc:302] T 247bddda55104f219ff5a26b18002466 P 53c5c6ea4be4420c93ba845b01cf4420 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53c5c6ea4be4420c93ba845b01cf4420, 6d8b211f4cdd433abc56965aa8d0b801; no voters:
I20260926 00:59:56.601991 7928 raft_consensus.cc:2781] T 247bddda55104f219ff5a26b18002466 P 53c5c6ea4be4420c93ba845b01cf4420 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 00:59:56.602046 7928 raft_consensus.cc:482] T 247bddda55104f219ff5a26b18002466 P 53c5c6ea4be4420c93ba845b01cf4420 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 00:59:56.602070 7928 raft_consensus.cc:3037] T 247bddda55104f219ff5a26b18002466 P 53c5c6ea4be4420c93ba845b01cf4420 [term 0 FOLLOWER]: Advancing to term 1
I20260926 00:59:56.602756 7928 raft_consensus.cc:504] T 247bddda55104f219ff5a26b18002466 P 53c5c6ea4be4420c93ba845b01cf4420 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "739b62f047fb4a3ba140ea7fc64d4112" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 34953 } } peers { permanent_uuid: "6d8b211f4cdd433abc56965aa8d0b801" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37333 } } peers { permanent_uuid: "53c5c6ea4be4420c93ba845b01cf4420" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 45281 } }
I20260926 00:59:56.602933 7928 leader_election.cc:288] T 247bddda55104f219ff5a26b18002466 P 53c5c6ea4be4420c93ba845b01cf4420 [CANDIDATE]: Term 1 election: Requested vote from peers 739b62f047fb4a3ba140ea7fc64d4112 (127.3.75.3:34953), 6d8b211f4cdd433abc56965aa8d0b801 (127.3.75.1:37333)
I20260926 00:59:56.603024 7858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "247bddda55104f219ff5a26b18002466" candidate_uuid: "53c5c6ea4be4420c93ba845b01cf4420" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "739b62f047fb4a3ba140ea7fc64d4112"
I20260926 00:59:56.603039 7592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "247bddda55104f219ff5a26b18002466" candidate_uuid: "53c5c6ea4be4420c93ba845b01cf4420" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d8b211f4cdd433abc56965aa8d0b801"
I20260926 00:59:56.603099 7858 raft_consensus.cc:3037] T 247bddda55104f219ff5a26b18002466 P 739b62f047fb4a3ba140ea7fc64d4112 [term 0 FOLLOWER]: Advancing to term 1
I20260926 00:59:56.603147 7592 raft_consensus.cc:3037] T 247bddda55104f219ff5a26b18002466 P 6d8b211f4cdd433abc56965aa8d0b801 [term 0 FOLLOWER]: Advancing to term 1
I20260926 00:59:56.603770 7858 raft_consensus.cc:2454] T 247bddda55104f219ff5a26b18002466 P 739b62f047fb4a3ba140ea7fc64d4112 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53c5c6ea4be4420c93ba845b01cf4420 in term 1.
I20260926 00:59:56.603901 7592 raft_consensus.cc:2454] T 247bddda55104f219ff5a26b18002466 P 6d8b211f4cdd433abc56965aa8d0b801 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53c5c6ea4be4420c93ba845b01cf4420 in term 1.
I20260926 00:59:56.603968 7659 leader_election.cc:302] T 247bddda55104f219ff5a26b18002466 P 53c5c6ea4be4420c93ba845b01cf4420 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53c5c6ea4be4420c93ba845b01cf4420, 739b62f047fb4a3ba140ea7fc64d4112; no voters:
I20260926 00:59:56.604077 7928 raft_consensus.cc:2781] T 247bddda55104f219ff5a26b18002466 P 53c5c6ea4be4420c93ba845b01cf4420 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 00:59:56.604247 7928 raft_consensus.cc:686] T 247bddda55104f219ff5a26b18002466 P 53c5c6ea4be4420c93ba845b01cf4420 [term 1 LEADER]: Becoming Leader. State: Replica: 53c5c6ea4be4420c93ba845b01cf4420, State: Running, Role: LEADER
I20260926 00:59:56.604359 7928 consensus_queue.cc:237] T 247bddda55104f219ff5a26b18002466 P 53c5c6ea4be4420c93ba845b01cf4420 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "739b62f047fb4a3ba140ea7fc64d4112" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 34953 } } peers { permanent_uuid: "6d8b211f4cdd433abc56965aa8d0b801" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37333 } } peers { permanent_uuid: "53c5c6ea4be4420c93ba845b01cf4420" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 45281 } }
I20260926 00:59:56.605177 7447 catalog_manager.cc:5495] T 247bddda55104f219ff5a26b18002466 P 53c5c6ea4be4420c93ba845b01cf4420 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53c5c6ea4be4420c93ba845b01cf4420 (127.3.75.2). New cstate: current_term: 1 leader_uuid: "53c5c6ea4be4420c93ba845b01cf4420" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "739b62f047fb4a3ba140ea7fc64d4112" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 34953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d8b211f4cdd433abc56965aa8d0b801" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53c5c6ea4be4420c93ba845b01cf4420" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 45281 } health_report { overall_health: HEALTHY } } }
I20260926 00:59:56.625836 7920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:56.623446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49200 (local address 127.3.75.2:45281)
0926 00:59:56.623511 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:56.623513 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:56.623535 (+ 22us) server_negotiation.cc:407] Connection header received
0926 00:59:56.623575 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:56.623576 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:56.623595 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:56.623670 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 00:59:56.623823 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:56.624279 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:56.625058 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:56.625246 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:56.625324 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:56.625492 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 00:59:56.625593 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 00:59:56.625688 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:56.625707 (+ 19us) server_negotiation.cc:299] Negotiation successful
0926 00:59:56.625727 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 00:59:56.627624 7858 raft_consensus.cc:1261] T 247bddda55104f219ff5a26b18002466 P 739b62f047fb4a3ba140ea7fc64d4112 [term 1 FOLLOWER]: Refusing update from remote peer 53c5c6ea4be4420c93ba845b01cf4420: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 00:59:56.627955 7592 raft_consensus.cc:1261] T 247bddda55104f219ff5a26b18002466 P 6d8b211f4cdd433abc56965aa8d0b801 [term 1 FOLLOWER]: Refusing update from remote peer 53c5c6ea4be4420c93ba845b01cf4420: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 00:59:56.627991 7928 consensus_queue.cc:1035] T 247bddda55104f219ff5a26b18002466 P 53c5c6ea4be4420c93ba845b01cf4420 [LEADER]: Connected to new peer: Peer: permanent_uuid: "739b62f047fb4a3ba140ea7fc64d4112" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 34953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 00:59:56.628257 7928 consensus_queue.cc:1035] T 247bddda55104f219ff5a26b18002466 P 53c5c6ea4be4420c93ba845b01cf4420 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d8b211f4cdd433abc56965aa8d0b801" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 00:59:56.630379 7934 mvcc.cc:205] Tried to move back new op lower bound from 7333414488584122368 to 7333414488492171264. Current Snapshot: MvccSnapshot[applied={T|T < 7333414488584122368}]
I20260926 00:59:56.630731 7935 mvcc.cc:205] Tried to move back new op lower bound from 7333414488584122368 to 7333414488492171264. Current Snapshot: MvccSnapshot[applied={T|T < 7333414488584122368}]
I20260926 00:59:56.631258 7936 mvcc.cc:205] Tried to move back new op lower bound from 7333414488584122368 to 7333414488492171264. Current Snapshot: MvccSnapshot[applied={T|T < 7333414488584122368}]
I20260926 00:59:56.632777 7448 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57278:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 00:59:56.632863 7448 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57278:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 00:59:56.633517 7448 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d8348538241e48e7b063dff79609ab15: Sending DeleteTablet for 3 replicas of tablet 247bddda55104f219ff5a26b18002466
I20260926 00:59:56.633847 7838 tablet_service.cc:1514] Processing DeleteTablet for tablet 247bddda55104f219ff5a26b18002466 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 00:59:56 UTC) from {username='kudu', principal='kudu/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:50818
I20260926 00:59:56.633885 7572 tablet_service.cc:1514] Processing DeleteTablet for tablet 247bddda55104f219ff5a26b18002466 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 00:59:56 UTC) from {username='kudu', principal='kudu/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:39090
I20260926 00:59:56.633939 7705 tablet_service.cc:1514] Processing DeleteTablet for tablet 247bddda55104f219ff5a26b18002466 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 00:59:56 UTC) from {username='kudu', principal='kudu/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:49194
I20260926 00:59:56.634087 7944 tablet_replica.cc:330] T 247bddda55104f219ff5a26b18002466 P 6d8b211f4cdd433abc56965aa8d0b801: stopping tablet replica
I20260926 00:59:56.634182 7944 raft_consensus.cc:2229] T 247bddda55104f219ff5a26b18002466 P 6d8b211f4cdd433abc56965aa8d0b801 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 00:59:56.634232 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 7510
I20260926 00:59:56.634910 7945 tablet_replica.cc:330] T 247bddda55104f219ff5a26b18002466 P 739b62f047fb4a3ba140ea7fc64d4112: stopping tablet replica
I20260926 00:59:56.635004 7945 raft_consensus.cc:2229] T 247bddda55104f219ff5a26b18002466 P 739b62f047fb4a3ba140ea7fc64d4112 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 00:59:56.635090 7945 raft_consensus.cc:2258] T 247bddda55104f219ff5a26b18002466 P 739b62f047fb4a3ba140ea7fc64d4112 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 00:59:56.636010 7945 ts_tablet_manager.cc:1905] T 247bddda55104f219ff5a26b18002466 P 739b62f047fb4a3ba140ea7fc64d4112: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 00:59:56.636245 7946 tablet_replica.cc:330] T 247bddda55104f219ff5a26b18002466 P 53c5c6ea4be4420c93ba845b01cf4420: stopping tablet replica
I20260926 00:59:56.636373 7946 raft_consensus.cc:2229] T 247bddda55104f219ff5a26b18002466 P 53c5c6ea4be4420c93ba845b01cf4420 [term 1 LEADER]: Raft consensus shutting down.
I20260926 00:59:56.636538 7946 raft_consensus.cc:2258] T 247bddda55104f219ff5a26b18002466 P 53c5c6ea4be4420c93ba845b01cf4420 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 00:59:56.637063 7946 ts_tablet_manager.cc:1905] T 247bddda55104f219ff5a26b18002466 P 53c5c6ea4be4420c93ba845b01cf4420: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 00:59:56.637940 7945 ts_tablet_manager.cc:1918] T 247bddda55104f219ff5a26b18002466 P 739b62f047fb4a3ba140ea7fc64d4112: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 00:59:56.638024 7945 log.cc:1196] T 247bddda55104f219ff5a26b18002466 P 739b62f047fb4a3ba140ea7fc64d4112: Deleting WAL directory at /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/ts-2/wal/wals/247bddda55104f219ff5a26b18002466
I20260926 00:59:56.638278 7945 ts_tablet_manager.cc:1939] T 247bddda55104f219ff5a26b18002466 P 739b62f047fb4a3ba140ea7fc64d4112: Deleting consensus metadata
I20260926 00:59:56.638665 7433 catalog_manager.cc:4841] TS 739b62f047fb4a3ba140ea7fc64d4112 (127.3.75.3:34953): tablet 247bddda55104f219ff5a26b18002466 (table test-table [id=691136cc760147c58ebc4131e5330790]) successfully deleted
I20260926 00:59:56.639151 7946 ts_tablet_manager.cc:1918] T 247bddda55104f219ff5a26b18002466 P 53c5c6ea4be4420c93ba845b01cf4420: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 00:59:56.639277 7946 log.cc:1196] T 247bddda55104f219ff5a26b18002466 P 53c5c6ea4be4420c93ba845b01cf4420: Deleting WAL directory at /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384372073861-3372-0/minicluster-data/ts-1/wal/wals/247bddda55104f219ff5a26b18002466
I20260926 00:59:56.639710 7946 ts_tablet_manager.cc:1939] T 247bddda55104f219ff5a26b18002466 P 53c5c6ea4be4420c93ba845b01cf4420: Deleting consensus metadata
I20260926 00:59:56.640120 7434 catalog_manager.cc:4841] TS 53c5c6ea4be4420c93ba845b01cf4420 (127.3.75.2:45281): tablet 247bddda55104f219ff5a26b18002466 (table test-table [id=691136cc760147c58ebc4131e5330790]) successfully deleted
W20260926 00:59:56.640511 7432 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.75.1:37333 (error 108)
I20260926 00:59:56.641032 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:56.640904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.1:37333 (local address 127.0.0.1:39102)
0926 00:59:56.640965 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:56.641000 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.1:37333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260926 00:59:56.641026 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 7643
W20260926 00:59:56.641144 7432 catalog_manager.cc:4584] TS 6d8b211f4cdd433abc56965aa8d0b801 (127.3.75.1:37333): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 247bddda55104f219ff5a26b18002466: Network error: Client connection negotiation failed: client connection to 127.3.75.1:37333: connect: Connection refused (error 111)
I20260926 00:59:56.645787 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 7776
I20260926 00:59:56.650197 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 7417
2026-09-26T00:59:56Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1970 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-26T00:59:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T00:59:56Z Disabled control of system clock
I20260926 00:59:56.677167 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:37151
--webserver_interface=127.3.75.62
--webserver_port=0
--builtin_ntp_servers=127.3.75.20:34087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.75.62:37151
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 00:59:56.756207 7951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 00:59:56.756409 7951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 00:59:56.756429 7951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 00:59:56.757479 7951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 00:59:56.757514 7951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 00:59:56.757526 7951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 00:59:56.757542 7951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 00:59:56.757555 7951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 00:59:56.758816 7951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:34087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.75.62:37151
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:37151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 00:59:56.759024 7951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 00:59:56.759258 7951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 00:59:56.763973 7959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:56.763978 7956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 00:59:56.764190 7951 server_base.cc:1034] running on GCE node
W20260926 00:59:56.764092 7957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 00:59:56.764467 7951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 00:59:56.764751 7951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 00:59:56.765915 7951 hybrid_clock.cc:648] HybridClock initialized: now 1790384396765906 us; error 42 us; skew 500 ppm
I20260926 00:59:56.767129 7951 webserver.cc:460] Webserver started at http://127.3.75.62:45451/ using document root <none> and password file <none>
I20260926 00:59:56.767386 7951 fs_manager.cc:359] Metadata directory not provided
I20260926 00:59:56.767455 7951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 00:59:56.767604 7951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 00:59:56.768610 7951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/master-0/data/instance:
uuid: "19c32584558244df87d607be4e9700b3"
format_stamp: "Formatted at 2026-09-26 00:59:56 on dist-test-slave-7v10"
server_key: "2b03a3de07b505dcf3c47f8b0d012609"
server_key_iv: "c8c76d4816ba86612810f7076347ade8"
server_key_version: "encryptionkey@0"
I20260926 00:59:56.768963 7951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/master-0/wal/instance:
uuid: "19c32584558244df87d607be4e9700b3"
format_stamp: "Formatted at 2026-09-26 00:59:56 on dist-test-slave-7v10"
server_key: "2b03a3de07b505dcf3c47f8b0d012609"
server_key_iv: "c8c76d4816ba86612810f7076347ade8"
server_key_version: "encryptionkey@0"
I20260926 00:59:56.770682 7951 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 00:59:56.771463 7965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:56.771636 7951 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:56.771729 7951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/master-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/master-0/wal
uuid: "19c32584558244df87d607be4e9700b3"
format_stamp: "Formatted at 2026-09-26 00:59:56 on dist-test-slave-7v10"
server_key: "2b03a3de07b505dcf3c47f8b0d012609"
server_key_iv: "c8c76d4816ba86612810f7076347ade8"
server_key_version: "encryptionkey@0"
I20260926 00:59:56.771816 7951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 00:59:56.775072 7951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 00:59:56.775336 7951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 00:59:56.775480 7951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 00:59:56.779280 7951 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.62:37151
I20260926 00:59:56.779321 8017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.62:37151 every 8 connection(s)
I20260926 00:59:56.779654 7951 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
I20260926 00:59:56.780321 8018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 00:59:56.781030 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 7951
I20260926 00:59:56.781150 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/master-0/wal/instance
I20260926 00:59:56.781369 3372 external_mini_cluster.cc:1428] Setting key 012989f42d9f2ff6d9ee55a1272b0c23
I20260926 00:59:56.783793 8021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:56.781894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42048 (local address 127.3.75.62:37151)
0926 00:59:56.782086 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:56.782093 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:56.782116 (+ 23us) server_negotiation.cc:407] Connection header received
0926 00:59:56.782528 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:56.782534 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:56.782572 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:56.782684 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:56.783353 (+ 669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:56.783356 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:56.783356 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 00:59:56.783360 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:56.783403 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 00:59:56.783412 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:56.783467 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:56.783551 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 00:59:56.783618 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":84,"threads_started":1}
I20260926 00:59:56.784297 8018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19c32584558244df87d607be4e9700b3: Bootstrap starting.
I20260926 00:59:56.784878 8018 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19c32584558244df87d607be4e9700b3: Neither blocks nor log segments found. Creating new log.
I20260926 00:59:56.785148 8018 log.cc:824] T 00000000000000000000000000000000 P 19c32584558244df87d607be4e9700b3: Log is configured to *not* fsync() on all Append() calls
I20260926 00:59:56.785789 8018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19c32584558244df87d607be4e9700b3: No bootstrap required, opened a new log
I20260926 00:59:56.787129 8018 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19c32584558244df87d607be4e9700b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19c32584558244df87d607be4e9700b3" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 37151 } }
I20260926 00:59:56.787277 8018 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19c32584558244df87d607be4e9700b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 00:59:56.787324 8018 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19c32584558244df87d607be4e9700b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19c32584558244df87d607be4e9700b3, State: Initialized, Role: FOLLOWER
I20260926 00:59:56.787460 8018 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19c32584558244df87d607be4e9700b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19c32584558244df87d607be4e9700b3" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 37151 } }
I20260926 00:59:56.787560 8018 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19c32584558244df87d607be4e9700b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 00:59:56.787606 8018 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19c32584558244df87d607be4e9700b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 00:59:56.787652 8018 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19c32584558244df87d607be4e9700b3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 00:59:56.788169 8018 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19c32584558244df87d607be4e9700b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19c32584558244df87d607be4e9700b3" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 37151 } }
I20260926 00:59:56.788290 8018 leader_election.cc:302] T 00000000000000000000000000000000 P 19c32584558244df87d607be4e9700b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19c32584558244df87d607be4e9700b3; no voters:
I20260926 00:59:56.788465 8018 leader_election.cc:288] T 00000000000000000000000000000000 P 19c32584558244df87d607be4e9700b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 00:59:56.788704 8018 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19c32584558244df87d607be4e9700b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 00:59:56.788766 8023 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19c32584558244df87d607be4e9700b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 00:59:56.788905 8023 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19c32584558244df87d607be4e9700b3 [term 1 LEADER]: Becoming Leader. State: Replica: 19c32584558244df87d607be4e9700b3, State: Running, Role: LEADER
I20260926 00:59:56.788995 8023 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19c32584558244df87d607be4e9700b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19c32584558244df87d607be4e9700b3" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 37151 } }
I20260926 00:59:56.789369 8024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19c32584558244df87d607be4e9700b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19c32584558244df87d607be4e9700b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19c32584558244df87d607be4e9700b3" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 37151 } } }
I20260926 00:59:56.789367 8023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19c32584558244df87d607be4e9700b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19c32584558244df87d607be4e9700b3. Latest consensus state: current_term: 1 leader_uuid: "19c32584558244df87d607be4e9700b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19c32584558244df87d607be4e9700b3" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 37151 } } }
I20260926 00:59:56.789486 8024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19c32584558244df87d607be4e9700b3 [sys.catalog]: This master's current role is: LEADER
I20260926 00:59:56.789512 8023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19c32584558244df87d607be4e9700b3 [sys.catalog]: This master's current role is: LEADER
I20260926 00:59:56.789666 8032 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 00:59:56.790128 8032 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 00:59:56.791652 8032 catalog_manager.cc:1347] Generated new cluster ID: 0beedd3908094ab28bfeae0e9197354b
I20260926 00:59:56.791704 8032 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 00:59:56.808234 8032 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 00:59:56.808400 8032 catalog_manager.cc:1504] Loading token signing keys...
I20260926 00:59:56.821638 8032 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19c32584558244df87d607be4e9700b3: Generated new TSK 0
I20260926 00:59:56.821818 8032 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 00:59:56.823273 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.1:0
--local_ip_for_outbound_sockets=127.3.75.1
--webserver_interface=127.3.75.1
--webserver_port=0
--tserver_master_addrs=127.3.75.62:37151
--builtin_ntp_servers=127.3.75.20:34087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 00:59:56.920010 8042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 00:59:56.920203 8042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 00:59:56.920225 8042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 00:59:56.921379 8042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 00:59:56.921419 8042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 00:59:56.921444 8042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.1
I20260926 00:59:56.922747 8042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:34087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.75.1
--webserver_port=0
--tserver_master_addrs=127.3.75.62:37151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.1
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 00:59:56.923020 8042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 00:59:56.923307 8042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 00:59:56.928249 8047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:56.928251 8050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:56.928244 8048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 00:59:56.929282 8042 server_base.cc:1034] running on GCE node
I20260926 00:59:56.929509 8042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 00:59:56.929761 8042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 00:59:56.930927 8042 hybrid_clock.cc:648] HybridClock initialized: now 1790384396930915 us; error 27 us; skew 500 ppm
I20260926 00:59:56.932215 8042 webserver.cc:460] Webserver started at http://127.3.75.1:42827/ using document root <none> and password file <none>
I20260926 00:59:56.932475 8042 fs_manager.cc:359] Metadata directory not provided
I20260926 00:59:56.932539 8042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 00:59:56.932668 8042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 00:59:56.933693 8042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/ts-0/data/instance:
uuid: "91d4b6095aaa484ab11390a6d0622fe0"
format_stamp: "Formatted at 2026-09-26 00:59:56 on dist-test-slave-7v10"
server_key: "1c4003165fe250699c9f77ebf7c46ee9"
server_key_iv: "eca31e0e26f7a708819a9bf20fbb300f"
server_key_version: "encryptionkey@0"
I20260926 00:59:56.934046 8042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/ts-0/wal/instance:
uuid: "91d4b6095aaa484ab11390a6d0622fe0"
format_stamp: "Formatted at 2026-09-26 00:59:56 on dist-test-slave-7v10"
server_key: "1c4003165fe250699c9f77ebf7c46ee9"
server_key_iv: "eca31e0e26f7a708819a9bf20fbb300f"
server_key_version: "encryptionkey@0"
I20260926 00:59:56.935415 8042 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 00:59:56.936131 8056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:56.936297 8042 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 00:59:56.936403 8042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/ts-0/wal
uuid: "91d4b6095aaa484ab11390a6d0622fe0"
format_stamp: "Formatted at 2026-09-26 00:59:56 on dist-test-slave-7v10"
server_key: "1c4003165fe250699c9f77ebf7c46ee9"
server_key_iv: "eca31e0e26f7a708819a9bf20fbb300f"
server_key_version: "encryptionkey@0"
I20260926 00:59:56.936455 8042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 00:59:56.939710 8042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 00:59:56.939917 8042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 00:59:56.940011 8042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 00:59:56.940155 8042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 00:59:56.940402 8042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 00:59:56.940436 8042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:56.940455 8042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 00:59:56.940469 8042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:56.945827 8042 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.1:36871
I20260926 00:59:56.945904 8169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.1:36871 every 8 connection(s)
I20260926 00:59:56.946219 8042 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
I20260926 00:59:56.948424 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 8042
I20260926 00:59:56.948557 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/ts-0/wal/instance
I20260926 00:59:56.948719 3372 external_mini_cluster.cc:1428] Setting key 366a293c75c87a43b6b55dc1ddee44c3
I20260926 00:59:56.949512 8021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:56.946944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:42887 (local address 127.3.75.62:37151)
0926 00:59:56.947005 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:56.947007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:56.947246 (+ 239us) server_negotiation.cc:407] Connection header received
0926 00:59:56.947745 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:56.947746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:56.947752 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:56.947811 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:56.949183 (+ 1372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:56.949185 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:56.949185 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 00:59:56.949189 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:56.949204 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 00:59:56.949208 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:56.949270 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:56.949429 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 00:59:56.949444 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 00:59:56.949720 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.2:0
--local_ip_for_outbound_sockets=127.3.75.2
--webserver_interface=127.3.75.2
--webserver_port=0
--tserver_master_addrs=127.3.75.62:37151
--builtin_ntp_servers=127.3.75.20:34087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 00:59:56.949733 8172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:56.946930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:37151 (local address 127.3.75.1:42887)
0926 00:59:56.947123 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:56.947155 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:56.947638 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:56.947895 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:56.947899 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:56.947907 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 00:59:56.949010 (+ 1103us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 00:59:56.949011 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 00:59:56.949047 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 00:59:56.949081 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 00:59:56.949329 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 00:59:56.949335 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 00:59:56.949340 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 00:59:56.949439 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 00:59:56.949502 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":85,"threads_started":1}
I20260926 00:59:56.951261 8170 heartbeater.cc:347] Connected to a master server at 127.3.75.62:37151
I20260926 00:59:56.951354 8170 heartbeater.cc:464] Registering TS with master...
I20260926 00:59:56.951517 8170 heartbeater.cc:511] Master 127.3.75.62:37151 requested a full tablet report, sending...
I20260926 00:59:56.952064 7982 ts_manager.cc:194] Registered new tserver with Master: 91d4b6095aaa484ab11390a6d0622fe0 (127.3.75.1:36871)
W20260926 00:59:57.030771 8173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 00:59:57.031023 8173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 00:59:57.031047 8173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 00:59:57.032217 8173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 00:59:57.032253 8173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 00:59:57.032305 8173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.2
I20260926 00:59:57.033603 8173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:34087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.75.2
--webserver_port=0
--tserver_master_addrs=127.3.75.62:37151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.2
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 00:59:57.033826 8173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 00:59:57.034129 8173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 00:59:57.039048 8178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:57.039055 8179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:57.039059 8181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 00:59:57.039194 8173 server_base.cc:1034] running on GCE node
I20260926 00:59:57.039393 8173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 00:59:57.039661 8173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 00:59:57.040829 8173 hybrid_clock.cc:648] HybridClock initialized: now 1790384397040803 us; error 50 us; skew 500 ppm
I20260926 00:59:57.042055 8173 webserver.cc:460] Webserver started at http://127.3.75.2:44919/ using document root <none> and password file <none>
I20260926 00:59:57.042307 8173 fs_manager.cc:359] Metadata directory not provided
I20260926 00:59:57.042388 8173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 00:59:57.042546 8173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 00:59:57.043679 8173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/ts-1/data/instance:
uuid: "6898c300712547d38d7d31542982d8bd"
format_stamp: "Formatted at 2026-09-26 00:59:57 on dist-test-slave-7v10"
server_key: "0159ff0099ceb34c68519304815de157"
server_key_iv: "6d1e7fce5cf9dbf4031017a4bbcc4a37"
server_key_version: "encryptionkey@0"
I20260926 00:59:57.044057 8173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/ts-1/wal/instance:
uuid: "6898c300712547d38d7d31542982d8bd"
format_stamp: "Formatted at 2026-09-26 00:59:57 on dist-test-slave-7v10"
server_key: "0159ff0099ceb34c68519304815de157"
server_key_iv: "6d1e7fce5cf9dbf4031017a4bbcc4a37"
server_key_version: "encryptionkey@0"
I20260926 00:59:57.045399 8173 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 00:59:57.046067 8187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:57.046238 8173 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 00:59:57.046329 8173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/ts-1/wal
uuid: "6898c300712547d38d7d31542982d8bd"
format_stamp: "Formatted at 2026-09-26 00:59:57 on dist-test-slave-7v10"
server_key: "0159ff0099ceb34c68519304815de157"
server_key_iv: "6d1e7fce5cf9dbf4031017a4bbcc4a37"
server_key_version: "encryptionkey@0"
I20260926 00:59:57.046381 8173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 00:59:57.049602 8173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 00:59:57.049810 8173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 00:59:57.049911 8173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 00:59:57.050058 8173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 00:59:57.050283 8173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 00:59:57.050312 8173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:57.050330 8173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 00:59:57.050343 8173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:57.055950 8173 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.2:39081
I20260926 00:59:57.056006 8300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.2:39081 every 8 connection(s)
I20260926 00:59:57.056320 8173 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
I20260926 00:59:57.059257 8021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:57.057015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:45435 (local address 127.3.75.62:37151)
0926 00:59:57.057104 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:57.057107 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:57.057301 (+ 194us) server_negotiation.cc:407] Connection header received
0926 00:59:57.057751 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:57.057752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:57.057757 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:57.057811 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:57.059044 (+ 1233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:57.059046 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:57.059046 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 00:59:57.059050 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:57.059063 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 00:59:57.059067 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:57.059119 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:57.059190 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 00:59:57.059202 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 00:59:57.059446 8303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:57.056984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:37151 (local address 127.3.75.2:45435)
0926 00:59:57.057186 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:57.057217 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:57.057645 (+ 428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:57.057865 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:57.057868 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:57.057875 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0926 00:59:57.058801 (+ 926us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 00:59:57.058801 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 00:59:57.058828 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 00:59:57.058856 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 00:59:57.059118 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 00:59:57.059121 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 00:59:57.059124 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 00:59:57.059177 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 00:59:57.059223 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":108,"threads_started":1}
I20260926 00:59:57.059954 8301 heartbeater.cc:347] Connected to a master server at 127.3.75.62:37151
I20260926 00:59:57.060050 8301 heartbeater.cc:464] Registering TS with master...
I20260926 00:59:57.060195 8301 heartbeater.cc:511] Master 127.3.75.62:37151 requested a full tablet report, sending...
I20260926 00:59:57.060554 7982 ts_manager.cc:194] Registered new tserver with Master: 6898c300712547d38d7d31542982d8bd (127.3.75.2:39081)
I20260926 00:59:57.064451 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 8173
I20260926 00:59:57.064560 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/ts-1/wal/instance
I20260926 00:59:57.064743 3372 external_mini_cluster.cc:1428] Setting key 2b73d52ab3e49966427bb92eab77cb7d
I20260926 00:59:57.065608 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.3:0
--local_ip_for_outbound_sockets=127.3.75.3
--webserver_interface=127.3.75.3
--webserver_port=0
--tserver_master_addrs=127.3.75.62:37151
--builtin_ntp_servers=127.3.75.20:34087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 00:59:57.143596 8304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 00:59:57.143788 8304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 00:59:57.143810 8304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 00:59:57.144882 8304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 00:59:57.144917 8304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 00:59:57.144941 8304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.3
I20260926 00:59:57.146183 8304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:34087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.75.3
--webserver_port=0
--tserver_master_addrs=127.3.75.62:37151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.3
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 00:59:57.146361 8304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 00:59:57.146590 8304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 00:59:57.151443 8310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:57.151451 8309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:57.151458 8312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 00:59:57.151684 8304 server_base.cc:1034] running on GCE node
I20260926 00:59:57.151922 8304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 00:59:57.152163 8304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 00:59:57.153344 8304 hybrid_clock.cc:648] HybridClock initialized: now 1790384397153322 us; error 49 us; skew 500 ppm
I20260926 00:59:57.154484 8304 webserver.cc:460] Webserver started at http://127.3.75.3:34043/ using document root <none> and password file <none>
I20260926 00:59:57.154716 8304 fs_manager.cc:359] Metadata directory not provided
I20260926 00:59:57.154783 8304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 00:59:57.154949 8304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 00:59:57.155954 8304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/ts-2/data/instance:
uuid: "5829ecb68294410eb5019d05a9e48b91"
format_stamp: "Formatted at 2026-09-26 00:59:57 on dist-test-slave-7v10"
server_key: "63c8870c5222e78e4c71e1d271588f05"
server_key_iv: "8eb637c54f3bd581ebb846bc7f692b19"
server_key_version: "encryptionkey@0"
I20260926 00:59:57.156309 8304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/ts-2/wal/instance:
uuid: "5829ecb68294410eb5019d05a9e48b91"
format_stamp: "Formatted at 2026-09-26 00:59:57 on dist-test-slave-7v10"
server_key: "63c8870c5222e78e4c71e1d271588f05"
server_key_iv: "8eb637c54f3bd581ebb846bc7f692b19"
server_key_version: "encryptionkey@0"
I20260926 00:59:57.157552 8304 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 00:59:57.158206 8318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:57.158370 8304 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 00:59:57.158464 8304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/ts-2/wal
uuid: "5829ecb68294410eb5019d05a9e48b91"
format_stamp: "Formatted at 2026-09-26 00:59:57 on dist-test-slave-7v10"
server_key: "63c8870c5222e78e4c71e1d271588f05"
server_key_iv: "8eb637c54f3bd581ebb846bc7f692b19"
server_key_version: "encryptionkey@0"
I20260926 00:59:57.158540 8304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 00:59:57.161787 8304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 00:59:57.162032 8304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 00:59:57.162168 8304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 00:59:57.162379 8304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 00:59:57.162670 8304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 00:59:57.162724 8304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:57.162760 8304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 00:59:57.162788 8304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:57.168401 8304 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.3:37829
I20260926 00:59:57.168447 8431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.3:37829 every 8 connection(s)
I20260926 00:59:57.168721 8304 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
I20260926 00:59:57.169867 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 8304
I20260926 00:59:57.169955 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/ts-2/wal/instance
I20260926 00:59:57.170145 3372 external_mini_cluster.cc:1428] Setting key 49e2ad267808cda4665bcbf85b72a52f
I20260926 00:59:57.171665 8021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:57.169328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.3:42983 (local address 127.3.75.62:37151)
0926 00:59:57.169393 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:57.169394 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:57.169604 (+ 210us) server_negotiation.cc:407] Connection header received
0926 00:59:57.170083 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:57.170085 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:57.170092 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:57.170136 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:57.171435 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:57.171436 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:57.171436 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 00:59:57.171441 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:57.171455 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 00:59:57.171459 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:57.171512 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:57.171587 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 00:59:57.171600 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 00:59:57.171859 8434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:57.169326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:37151 (local address 127.3.75.3:42983)
0926 00:59:57.169512 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:57.169541 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:57.169983 (+ 442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:57.170209 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:57.170213 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:57.170223 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 00:59:57.171286 (+ 1063us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 00:59:57.171287 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 00:59:57.171312 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 00:59:57.171339 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 00:59:57.171512 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 00:59:57.171517 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 00:59:57.171519 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 00:59:57.171595 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 00:59:57.171650 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":101,"threads_started":1}
I20260926 00:59:57.172282 8432 heartbeater.cc:347] Connected to a master server at 127.3.75.62:37151
I20260926 00:59:57.172369 8432 heartbeater.cc:464] Registering TS with master...
I20260926 00:59:57.172468 8432 heartbeater.cc:511] Master 127.3.75.62:37151 requested a full tablet report, sending...
I20260926 00:59:57.172784 7982 ts_manager.cc:194] Registered new tserver with Master: 5829ecb68294410eb5019d05a9e48b91 (127.3.75.3:37829)
I20260926 00:59:57.181722 3372 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 00:59:57.184056 8021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:57.182979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42064 (local address 127.3.75.62:37151)
0926 00:59:57.183038 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:57.183040 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:57.183179 (+ 139us) server_negotiation.cc:407] Connection header received
0926 00:59:57.183227 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:57.183228 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:57.183232 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:57.183278 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:57.183861 (+ 583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:57.183862 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:57.183862 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 00:59:57.183865 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:57.183874 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 00:59:57.183877 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:57.183913 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:57.183949 (+ 36us) server_negotiation.cc:299] Negotiation successful
0926 00:59:57.183959 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 00:59:57.185644 7982 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42064:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 00:59:57.187196 7982 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 00:59:57.191097 8442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:57.189318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.3:37829 (local address 127.0.0.1:39634)
0926 00:59:57.189597 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:57.189636 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:57.189688 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:57.190004 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:57.190011 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:57.190017 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0926 00:59:57.190484 (+ 467us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 00:59:57.190485 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 00:59:57.190501 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 00:59:57.190523 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 00:59:57.190951 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 00:59:57.190957 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 00:59:57.190960 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 00:59:57.191027 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 00:59:57.191039 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":86,"threads_started":1}
I20260926 00:59:57.191218 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:57.189318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.2:39081 (local address 127.0.0.1:33808)
0926 00:59:57.189812 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:57.189821 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:57.189870 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:57.190303 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:57.190303 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:57.190305 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0926 00:59:57.190788 (+ 483us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 00:59:57.190789 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 00:59:57.190795 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 00:59:57.190806 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 00:59:57.191087 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 00:59:57.191088 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 00:59:57.191089 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 00:59:57.191133 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 00:59:57.191145 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"mutex_wait_us":2,"thread_start_us":90,"threads_started":1}
I20260926 00:59:57.191263 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:57.189561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39634 (local address 127.3.75.3:37829)
0926 00:59:57.189715 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:57.189719 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:57.189731 (+ 12us) server_negotiation.cc:407] Connection header received
0926 00:59:57.189776 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:57.189778 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:57.189793 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:57.189998 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:57.190774 (+ 776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:57.190777 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:57.190778 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 00:59:57.190782 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:57.190812 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 00:59:57.190826 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:57.190922 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:57.191057 (+ 135us) server_negotiation.cc:299] Negotiation successful
0926 00:59:57.191075 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":96,"threads_started":1}
I20260926 00:59:57.191479 8445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:57.189561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33808 (local address 127.3.75.2:39081)
0926 00:59:57.189754 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:57.189758 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:57.189928 (+ 170us) server_negotiation.cc:407] Connection header received
0926 00:59:57.189956 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:57.189958 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:57.189967 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:57.190034 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:57.190915 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:57.190924 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:57.190924 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 00:59:57.190928 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:57.190960 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 00:59:57.190988 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:57.191061 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:57.191378 (+ 317us) server_negotiation.cc:299] Negotiation successful
0926 00:59:57.191396 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":86,"threads_started":1}
I20260926 00:59:57.191906 8366 tablet_service.cc:1467] Processing CreateTablet for tablet 8d8455a33dcb413197adc0d69d9d9acf (DEFAULT_TABLE table=test-table [id=98566b4ca8bb41579ed8e9ce5bccf216]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 00:59:57.192040 8440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:57.189319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.1:36871 (local address 127.0.0.1:45438)
0926 00:59:57.189806 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:57.189816 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:57.189856 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:57.190630 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:57.190631 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:57.190634 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0926 00:59:57.191621 (+ 987us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 00:59:57.191621 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 00:59:57.191627 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 00:59:57.191639 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 00:59:57.191916 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 00:59:57.191917 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 00:59:57.191918 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 00:59:57.191963 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 00:59:57.191976 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":68,"threads_started":1}
I20260926 00:59:57.192063 8235 tablet_service.cc:1467] Processing CreateTablet for tablet 8d8455a33dcb413197adc0d69d9d9acf (DEFAULT_TABLE table=test-table [id=98566b4ca8bb41579ed8e9ce5bccf216]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 00:59:57.192194 8443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:57.189406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45438 (local address 127.3.75.1:36871)
0926 00:59:57.190478 (+ 1072us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:57.190482 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:57.190489 (+ 7us) server_negotiation.cc:407] Connection header received
0926 00:59:57.190509 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:57.190511 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:57.190545 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:57.190616 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:57.191755 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:57.191766 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:57.191766 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 00:59:57.191771 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:57.191805 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 00:59:57.191835 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:57.191892 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:57.192124 (+ 232us) server_negotiation.cc:299] Negotiation successful
0926 00:59:57.192137 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1039,"thread_start_us":85,"threads_started":1}
I20260926 00:59:57.192251 8366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d8455a33dcb413197adc0d69d9d9acf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 00:59:57.192322 8235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d8455a33dcb413197adc0d69d9d9acf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 00:59:57.195050 8104 tablet_service.cc:1467] Processing CreateTablet for tablet 8d8455a33dcb413197adc0d69d9d9acf (DEFAULT_TABLE table=test-table [id=98566b4ca8bb41579ed8e9ce5bccf216]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 00:59:57.195372 8104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d8455a33dcb413197adc0d69d9d9acf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 00:59:57.195568 8447 tablet_bootstrap.cc:492] T 8d8455a33dcb413197adc0d69d9d9acf P 6898c300712547d38d7d31542982d8bd: Bootstrap starting.
I20260926 00:59:57.196209 8447 tablet_bootstrap.cc:654] T 8d8455a33dcb413197adc0d69d9d9acf P 6898c300712547d38d7d31542982d8bd: Neither blocks nor log segments found. Creating new log.
I20260926 00:59:57.196537 8447 log.cc:824] T 8d8455a33dcb413197adc0d69d9d9acf P 6898c300712547d38d7d31542982d8bd: Log is configured to *not* fsync() on all Append() calls
I20260926 00:59:57.196841 8446 tablet_bootstrap.cc:492] T 8d8455a33dcb413197adc0d69d9d9acf P 5829ecb68294410eb5019d05a9e48b91: Bootstrap starting.
I20260926 00:59:57.197283 8447 tablet_bootstrap.cc:492] T 8d8455a33dcb413197adc0d69d9d9acf P 6898c300712547d38d7d31542982d8bd: No bootstrap required, opened a new log
I20260926 00:59:57.197368 8447 ts_tablet_manager.cc:1397] T 8d8455a33dcb413197adc0d69d9d9acf P 6898c300712547d38d7d31542982d8bd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 00:59:57.197480 8446 tablet_bootstrap.cc:654] T 8d8455a33dcb413197adc0d69d9d9acf P 5829ecb68294410eb5019d05a9e48b91: Neither blocks nor log segments found. Creating new log.
I20260926 00:59:57.197788 8446 log.cc:824] T 8d8455a33dcb413197adc0d69d9d9acf P 5829ecb68294410eb5019d05a9e48b91: Log is configured to *not* fsync() on all Append() calls
I20260926 00:59:57.198568 8446 tablet_bootstrap.cc:492] T 8d8455a33dcb413197adc0d69d9d9acf P 5829ecb68294410eb5019d05a9e48b91: No bootstrap required, opened a new log
I20260926 00:59:57.198642 8446 ts_tablet_manager.cc:1397] T 8d8455a33dcb413197adc0d69d9d9acf P 5829ecb68294410eb5019d05a9e48b91: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 00:59:57.199033 8450 tablet_bootstrap.cc:492] T 8d8455a33dcb413197adc0d69d9d9acf P 91d4b6095aaa484ab11390a6d0622fe0: Bootstrap starting.
I20260926 00:59:57.199654 8450 tablet_bootstrap.cc:654] T 8d8455a33dcb413197adc0d69d9d9acf P 91d4b6095aaa484ab11390a6d0622fe0: Neither blocks nor log segments found. Creating new log.
I20260926 00:59:57.199656 8447 raft_consensus.cc:348] T 8d8455a33dcb413197adc0d69d9d9acf P 6898c300712547d38d7d31542982d8bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d4b6095aaa484ab11390a6d0622fe0" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 36871 } } peers { permanent_uuid: "6898c300712547d38d7d31542982d8bd" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39081 } } peers { permanent_uuid: "5829ecb68294410eb5019d05a9e48b91" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 37829 } }
I20260926 00:59:57.199841 8447 raft_consensus.cc:374] T 8d8455a33dcb413197adc0d69d9d9acf P 6898c300712547d38d7d31542982d8bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 00:59:57.199885 8447 raft_consensus.cc:729] T 8d8455a33dcb413197adc0d69d9d9acf P 6898c300712547d38d7d31542982d8bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6898c300712547d38d7d31542982d8bd, State: Initialized, Role: FOLLOWER
I20260926 00:59:57.199962 8450 log.cc:824] T 8d8455a33dcb413197adc0d69d9d9acf P 91d4b6095aaa484ab11390a6d0622fe0: Log is configured to *not* fsync() on all Append() calls
I20260926 00:59:57.199998 8447 consensus_queue.cc:260] T 8d8455a33dcb413197adc0d69d9d9acf P 6898c300712547d38d7d31542982d8bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d4b6095aaa484ab11390a6d0622fe0" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 36871 } } peers { permanent_uuid: "6898c300712547d38d7d31542982d8bd" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39081 } } peers { permanent_uuid: "5829ecb68294410eb5019d05a9e48b91" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 37829 } }
I20260926 00:59:57.200259 8301 heartbeater.cc:503] Master 127.3.75.62:37151 was elected leader, sending a full tablet report...
I20260926 00:59:57.200516 8447 ts_tablet_manager.cc:1428] T 8d8455a33dcb413197adc0d69d9d9acf P 6898c300712547d38d7d31542982d8bd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 00:59:57.200699 8450 tablet_bootstrap.cc:492] T 8d8455a33dcb413197adc0d69d9d9acf P 91d4b6095aaa484ab11390a6d0622fe0: No bootstrap required, opened a new log
I20260926 00:59:57.200753 8450 ts_tablet_manager.cc:1397] T 8d8455a33dcb413197adc0d69d9d9acf P 91d4b6095aaa484ab11390a6d0622fe0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 00:59:57.201502 8446 raft_consensus.cc:348] T 8d8455a33dcb413197adc0d69d9d9acf P 5829ecb68294410eb5019d05a9e48b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d4b6095aaa484ab11390a6d0622fe0" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 36871 } } peers { permanent_uuid: "6898c300712547d38d7d31542982d8bd" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39081 } } peers { permanent_uuid: "5829ecb68294410eb5019d05a9e48b91" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 37829 } }
I20260926 00:59:57.201617 8446 raft_consensus.cc:374] T 8d8455a33dcb413197adc0d69d9d9acf P 5829ecb68294410eb5019d05a9e48b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 00:59:57.201637 8446 raft_consensus.cc:729] T 8d8455a33dcb413197adc0d69d9d9acf P 5829ecb68294410eb5019d05a9e48b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5829ecb68294410eb5019d05a9e48b91, State: Initialized, Role: FOLLOWER
I20260926 00:59:57.201737 8446 consensus_queue.cc:260] T 8d8455a33dcb413197adc0d69d9d9acf P 5829ecb68294410eb5019d05a9e48b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d4b6095aaa484ab11390a6d0622fe0" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 36871 } } peers { permanent_uuid: "6898c300712547d38d7d31542982d8bd" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39081 } } peers { permanent_uuid: "5829ecb68294410eb5019d05a9e48b91" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 37829 } }
I20260926 00:59:57.201972 8446 ts_tablet_manager.cc:1428] T 8d8455a33dcb413197adc0d69d9d9acf P 5829ecb68294410eb5019d05a9e48b91: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 00:59:57.202015 8432 heartbeater.cc:503] Master 127.3.75.62:37151 was elected leader, sending a full tablet report...
I20260926 00:59:57.203157 8450 raft_consensus.cc:348] T 8d8455a33dcb413197adc0d69d9d9acf P 91d4b6095aaa484ab11390a6d0622fe0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d4b6095aaa484ab11390a6d0622fe0" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 36871 } } peers { permanent_uuid: "6898c300712547d38d7d31542982d8bd" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39081 } } peers { permanent_uuid: "5829ecb68294410eb5019d05a9e48b91" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 37829 } }
I20260926 00:59:57.203267 8450 raft_consensus.cc:374] T 8d8455a33dcb413197adc0d69d9d9acf P 91d4b6095aaa484ab11390a6d0622fe0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 00:59:57.203287 8450 raft_consensus.cc:729] T 8d8455a33dcb413197adc0d69d9d9acf P 91d4b6095aaa484ab11390a6d0622fe0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d4b6095aaa484ab11390a6d0622fe0, State: Initialized, Role: FOLLOWER
I20260926 00:59:57.203382 8450 consensus_queue.cc:260] T 8d8455a33dcb413197adc0d69d9d9acf P 91d4b6095aaa484ab11390a6d0622fe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d4b6095aaa484ab11390a6d0622fe0" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 36871 } } peers { permanent_uuid: "6898c300712547d38d7d31542982d8bd" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39081 } } peers { permanent_uuid: "5829ecb68294410eb5019d05a9e48b91" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 37829 } }
I20260926 00:59:57.203598 8450 ts_tablet_manager.cc:1428] T 8d8455a33dcb413197adc0d69d9d9acf P 91d4b6095aaa484ab11390a6d0622fe0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 00:59:57.203715 8170 heartbeater.cc:503] Master 127.3.75.62:37151 was elected leader, sending a full tablet report...
I20260926 00:59:57.229377 8452 raft_consensus.cc:482] T 8d8455a33dcb413197adc0d69d9d9acf P 6898c300712547d38d7d31542982d8bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 00:59:57.229532 8452 raft_consensus.cc:504] T 8d8455a33dcb413197adc0d69d9d9acf P 6898c300712547d38d7d31542982d8bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d4b6095aaa484ab11390a6d0622fe0" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 36871 } } peers { permanent_uuid: "6898c300712547d38d7d31542982d8bd" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39081 } } peers { permanent_uuid: "5829ecb68294410eb5019d05a9e48b91" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 37829 } }
I20260926 00:59:57.229918 8452 leader_election.cc:288] T 8d8455a33dcb413197adc0d69d9d9acf P 6898c300712547d38d7d31542982d8bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91d4b6095aaa484ab11390a6d0622fe0 (127.3.75.1:36871), 5829ecb68294410eb5019d05a9e48b91 (127.3.75.3:37829)
I20260926 00:59:57.231186 8303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:57.230097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.1:36871 (local address 127.3.75.2:50339)
0926 00:59:57.230174 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:57.230186 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:57.230226 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:57.230405 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:57.230406 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:57.230409 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0926 00:59:57.230818 (+ 409us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 00:59:57.230818 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 00:59:57.230824 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 00:59:57.230839 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 00:59:57.231038 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 00:59:57.231039 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 00:59:57.231040 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 00:59:57.231091 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 00:59:57.231105 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 00:59:57.231212 8443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:57.230181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:50339 (local address 127.3.75.1:36871)
0926 00:59:57.230253 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:57.230255 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:57.230265 (+ 10us) server_negotiation.cc:407] Connection header received
0926 00:59:57.230298 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:57.230299 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:57.230305 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:57.230398 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:57.230940 (+ 542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:57.230942 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:57.230943 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 00:59:57.230947 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:57.230966 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 00:59:57.230971 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:57.231027 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:57.231111 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 00:59:57.231126 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 00:59:57.231505 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:57.230178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:40603 (local address 127.3.75.3:37829)
0926 00:59:57.230259 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:57.230261 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:57.230445 (+ 184us) server_negotiation.cc:407] Connection header received
0926 00:59:57.230507 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:57.230508 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:57.230513 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:57.230570 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:57.231113 (+ 543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:57.231115 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:57.231115 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 00:59:57.231120 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:57.231134 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 00:59:57.231139 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:57.231193 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:57.231424 (+ 231us) server_negotiation.cc:299] Negotiation successful
0926 00:59:57.231440 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 00:59:57.231535 8124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8455a33dcb413197adc0d69d9d9acf" candidate_uuid: "6898c300712547d38d7d31542982d8bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d4b6095aaa484ab11390a6d0622fe0" is_pre_election: true
I20260926 00:59:57.231698 8455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:57.230100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.3:37829 (local address 127.3.75.2:40603)
0926 00:59:57.230380 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:57.230387 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:57.230445 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:57.230607 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:57.230608 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:57.230619 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0926 00:59:57.230996 (+ 377us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 00:59:57.230996 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 00:59:57.231019 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 00:59:57.231031 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 00:59:57.231198 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 00:59:57.231198 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 00:59:57.231199 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 00:59:57.231592 (+ 393us) client_negotiation.cc:240] Negotiation successful
0926 00:59:57.231607 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":111,"thread_start_us":109,"threads_started":1}
I20260926 00:59:57.231693 8124 raft_consensus.cc:2454] T 8d8455a33dcb413197adc0d69d9d9acf P 91d4b6095aaa484ab11390a6d0622fe0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6898c300712547d38d7d31542982d8bd in term 0.
I20260926 00:59:57.232021 8386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8455a33dcb413197adc0d69d9d9acf" candidate_uuid: "6898c300712547d38d7d31542982d8bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5829ecb68294410eb5019d05a9e48b91" is_pre_election: true
I20260926 00:59:57.232053 8189 leader_election.cc:302] T 8d8455a33dcb413197adc0d69d9d9acf P 6898c300712547d38d7d31542982d8bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6898c300712547d38d7d31542982d8bd, 91d4b6095aaa484ab11390a6d0622fe0; no voters:
I20260926 00:59:57.232157 8386 raft_consensus.cc:2454] T 8d8455a33dcb413197adc0d69d9d9acf P 5829ecb68294410eb5019d05a9e48b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6898c300712547d38d7d31542982d8bd in term 0.
I20260926 00:59:57.232159 8452 raft_consensus.cc:2781] T 8d8455a33dcb413197adc0d69d9d9acf P 6898c300712547d38d7d31542982d8bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 00:59:57.232213 8452 raft_consensus.cc:482] T 8d8455a33dcb413197adc0d69d9d9acf P 6898c300712547d38d7d31542982d8bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 00:59:57.232236 8452 raft_consensus.cc:3037] T 8d8455a33dcb413197adc0d69d9d9acf P 6898c300712547d38d7d31542982d8bd [term 0 FOLLOWER]: Advancing to term 1
I20260926 00:59:57.232986 8452 raft_consensus.cc:504] T 8d8455a33dcb413197adc0d69d9d9acf P 6898c300712547d38d7d31542982d8bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d4b6095aaa484ab11390a6d0622fe0" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 36871 } } peers { permanent_uuid: "6898c300712547d38d7d31542982d8bd" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39081 } } peers { permanent_uuid: "5829ecb68294410eb5019d05a9e48b91" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 37829 } }
I20260926 00:59:57.233124 8452 leader_election.cc:288] T 8d8455a33dcb413197adc0d69d9d9acf P 6898c300712547d38d7d31542982d8bd [CANDIDATE]: Term 1 election: Requested vote from peers 91d4b6095aaa484ab11390a6d0622fe0 (127.3.75.1:36871), 5829ecb68294410eb5019d05a9e48b91 (127.3.75.3:37829)
I20260926 00:59:57.233258 8124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8455a33dcb413197adc0d69d9d9acf" candidate_uuid: "6898c300712547d38d7d31542982d8bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d4b6095aaa484ab11390a6d0622fe0"
I20260926 00:59:57.233331 8386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8455a33dcb413197adc0d69d9d9acf" candidate_uuid: "6898c300712547d38d7d31542982d8bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5829ecb68294410eb5019d05a9e48b91"
I20260926 00:59:57.233342 8124 raft_consensus.cc:3037] T 8d8455a33dcb413197adc0d69d9d9acf P 91d4b6095aaa484ab11390a6d0622fe0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 00:59:57.233440 8386 raft_consensus.cc:3037] T 8d8455a33dcb413197adc0d69d9d9acf P 5829ecb68294410eb5019d05a9e48b91 [term 0 FOLLOWER]: Advancing to term 1
I20260926 00:59:57.234057 8124 raft_consensus.cc:2454] T 8d8455a33dcb413197adc0d69d9d9acf P 91d4b6095aaa484ab11390a6d0622fe0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6898c300712547d38d7d31542982d8bd in term 1.
I20260926 00:59:57.234166 8386 raft_consensus.cc:2454] T 8d8455a33dcb413197adc0d69d9d9acf P 5829ecb68294410eb5019d05a9e48b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6898c300712547d38d7d31542982d8bd in term 1.
I20260926 00:59:57.234213 8189 leader_election.cc:302] T 8d8455a33dcb413197adc0d69d9d9acf P 6898c300712547d38d7d31542982d8bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6898c300712547d38d7d31542982d8bd, 91d4b6095aaa484ab11390a6d0622fe0; no voters:
I20260926 00:59:57.234345 8452 raft_consensus.cc:2781] T 8d8455a33dcb413197adc0d69d9d9acf P 6898c300712547d38d7d31542982d8bd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 00:59:57.234512 8452 raft_consensus.cc:686] T 8d8455a33dcb413197adc0d69d9d9acf P 6898c300712547d38d7d31542982d8bd [term 1 LEADER]: Becoming Leader. State: Replica: 6898c300712547d38d7d31542982d8bd, State: Running, Role: LEADER
I20260926 00:59:57.234601 8452 consensus_queue.cc:237] T 8d8455a33dcb413197adc0d69d9d9acf P 6898c300712547d38d7d31542982d8bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d4b6095aaa484ab11390a6d0622fe0" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 36871 } } peers { permanent_uuid: "6898c300712547d38d7d31542982d8bd" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39081 } } peers { permanent_uuid: "5829ecb68294410eb5019d05a9e48b91" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 37829 } }
I20260926 00:59:57.235353 7982 catalog_manager.cc:5495] T 8d8455a33dcb413197adc0d69d9d9acf P 6898c300712547d38d7d31542982d8bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 6898c300712547d38d7d31542982d8bd (127.3.75.2). New cstate: current_term: 1 leader_uuid: "6898c300712547d38d7d31542982d8bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d4b6095aaa484ab11390a6d0622fe0" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 36871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6898c300712547d38d7d31542982d8bd" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5829ecb68294410eb5019d05a9e48b91" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 37829 } health_report { overall_health: UNKNOWN } } }
I20260926 00:59:57.249150 8445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:57.248300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33824 (local address 127.3.75.2:39081)
0926 00:59:57.248344 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:57.248350 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:57.248357 (+ 7us) server_negotiation.cc:407] Connection header received
0926 00:59:57.248398 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:57.248399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:57.248407 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:57.248462 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:57.249004 (+ 542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:57.249004 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:57.249005 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 00:59:57.249007 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:57.249017 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 00:59:57.249020 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:57.249057 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:57.249084 (+ 27us) server_negotiation.cc:299] Negotiation successful
0926 00:59:57.249093 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 00:59:57.250710 8124 raft_consensus.cc:1261] T 8d8455a33dcb413197adc0d69d9d9acf P 91d4b6095aaa484ab11390a6d0622fe0 [term 1 FOLLOWER]: Refusing update from remote peer 6898c300712547d38d7d31542982d8bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 00:59:57.250731 8386 raft_consensus.cc:1261] T 8d8455a33dcb413197adc0d69d9d9acf P 5829ecb68294410eb5019d05a9e48b91 [term 1 FOLLOWER]: Refusing update from remote peer 6898c300712547d38d7d31542982d8bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 00:59:57.251037 8452 consensus_queue.cc:1035] T 8d8455a33dcb413197adc0d69d9d9acf P 6898c300712547d38d7d31542982d8bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d4b6095aaa484ab11390a6d0622fe0" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 36871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 00:59:57.251161 8456 consensus_queue.cc:1035] T 8d8455a33dcb413197adc0d69d9d9acf P 6898c300712547d38d7d31542982d8bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5829ecb68294410eb5019d05a9e48b91" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 00:59:57.254305 8460 mvcc.cc:205] Tried to move back new op lower bound from 7333414491136569344 to 7333414491073454080. Current Snapshot: MvccSnapshot[applied={T|T < 7333414491136569344}]
I20260926 00:59:57.255673 7982 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 00:59:57.255753 7982 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 00:59:57.256448 7982 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 19c32584558244df87d607be4e9700b3: Sending DeleteTablet for 3 replicas of tablet 8d8455a33dcb413197adc0d69d9d9acf
I20260926 00:59:57.256762 8235 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d8455a33dcb413197adc0d69d9d9acf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 00:59:57 UTC) from {username='slave'} at 127.0.0.1:33808
I20260926 00:59:57.256781 8104 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d8455a33dcb413197adc0d69d9d9acf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 00:59:57 UTC) from {username='slave'} at 127.0.0.1:45438
I20260926 00:59:57.256965 8366 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d8455a33dcb413197adc0d69d9d9acf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 00:59:57 UTC) from {username='slave'} at 127.0.0.1:39634
I20260926 00:59:57.256991 8469 tablet_replica.cc:330] T 8d8455a33dcb413197adc0d69d9d9acf P 6898c300712547d38d7d31542982d8bd: stopping tablet replica
I20260926 00:59:57.257081 8469 raft_consensus.cc:2229] T 8d8455a33dcb413197adc0d69d9d9acf P 6898c300712547d38d7d31542982d8bd [term 1 LEADER]: Raft consensus shutting down.
I20260926 00:59:57.257159 8469 raft_consensus.cc:2258] T 8d8455a33dcb413197adc0d69d9d9acf P 6898c300712547d38d7d31542982d8bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 00:59:57.257174 8470 tablet_replica.cc:330] T 8d8455a33dcb413197adc0d69d9d9acf P 91d4b6095aaa484ab11390a6d0622fe0: stopping tablet replica
I20260926 00:59:57.257247 8470 raft_consensus.cc:2229] T 8d8455a33dcb413197adc0d69d9d9acf P 91d4b6095aaa484ab11390a6d0622fe0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 00:59:57.257252 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 8042
I20260926 00:59:57.258123 8469 ts_tablet_manager.cc:1905] T 8d8455a33dcb413197adc0d69d9d9acf P 6898c300712547d38d7d31542982d8bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 00:59:57.258728 8471 tablet_replica.cc:330] T 8d8455a33dcb413197adc0d69d9d9acf P 5829ecb68294410eb5019d05a9e48b91: stopping tablet replica
I20260926 00:59:57.258901 8471 raft_consensus.cc:2229] T 8d8455a33dcb413197adc0d69d9d9acf P 5829ecb68294410eb5019d05a9e48b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 00:59:57.259052 8471 raft_consensus.cc:2258] T 8d8455a33dcb413197adc0d69d9d9acf P 5829ecb68294410eb5019d05a9e48b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 00:59:57.259500 8471 ts_tablet_manager.cc:1905] T 8d8455a33dcb413197adc0d69d9d9acf P 5829ecb68294410eb5019d05a9e48b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 00:59:57.259654 8469 ts_tablet_manager.cc:1918] T 8d8455a33dcb413197adc0d69d9d9acf P 6898c300712547d38d7d31542982d8bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 00:59:57.259725 8469 log.cc:1196] T 8d8455a33dcb413197adc0d69d9d9acf P 6898c300712547d38d7d31542982d8bd: Deleting WAL directory at /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/ts-1/wal/wals/8d8455a33dcb413197adc0d69d9d9acf
I20260926 00:59:57.259982 8469 ts_tablet_manager.cc:1939] T 8d8455a33dcb413197adc0d69d9d9acf P 6898c300712547d38d7d31542982d8bd: Deleting consensus metadata
I20260926 00:59:57.260319 7966 catalog_manager.cc:4841] TS 6898c300712547d38d7d31542982d8bd (127.3.75.2:39081): tablet 8d8455a33dcb413197adc0d69d9d9acf (table test-table [id=98566b4ca8bb41579ed8e9ce5bccf216]) successfully deleted
I20260926 00:59:57.261296 8471 ts_tablet_manager.cc:1918] T 8d8455a33dcb413197adc0d69d9d9acf P 5829ecb68294410eb5019d05a9e48b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 00:59:57.261431 8471 log.cc:1196] T 8d8455a33dcb413197adc0d69d9d9acf P 5829ecb68294410eb5019d05a9e48b91: Deleting WAL directory at /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384372073861-3372-0/minicluster-data/ts-2/wal/wals/8d8455a33dcb413197adc0d69d9d9acf
I20260926 00:59:57.261740 8471 ts_tablet_manager.cc:1939] T 8d8455a33dcb413197adc0d69d9d9acf P 5829ecb68294410eb5019d05a9e48b91: Deleting consensus metadata
I20260926 00:59:57.262167 7968 catalog_manager.cc:4841] TS 5829ecb68294410eb5019d05a9e48b91 (127.3.75.3:37829): tablet 8d8455a33dcb413197adc0d69d9d9acf (table test-table [id=98566b4ca8bb41579ed8e9ce5bccf216]) successfully deleted
W20260926 00:59:57.263250 7967 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.75.1:36871 (error 108)
I20260926 00:59:57.263768 8440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:57.263642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.1:36871 (local address 127.0.0.1:45454)
0926 00:59:57.263698 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:57.263738 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.1:36871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 00:59:57.263828 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 8173
W20260926 00:59:57.267431 7967 catalog_manager.cc:4584] TS 91d4b6095aaa484ab11390a6d0622fe0 (127.3.75.1:36871): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8d8455a33dcb413197adc0d69d9d9acf: Network error: Client connection negotiation failed: client connection to 127.3.75.1:36871: connect: Connection refused (error 111)
I20260926 00:59:57.270633 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 8304
I20260926 00:59:57.276885 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 7951
2026-09-26T00:59:57Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (625 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260926 00:59:57.296082 3372 mini_oidc.cc:152] Starting JWKS server
I20260926 00:59:57.297181 3372 webserver.cc:460] Webserver started at https://127.0.0.1:40161/ using document root <none> and password file <none>
I20260926 00:59:57.297283 3372 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 00:59:57.297488 3372 webserver.cc:460] Webserver started at http://127.0.0.1:39325/ using document root <none> and password file <none>
2026-09-26T00:59:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T00:59:57Z Disabled control of system clock
I20260926 00:59:57.305088 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790384372073861-3372-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790384372073861-3372-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790384372073861-3372-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:37753
--webserver_interface=127.3.75.62
--webserver_port=0
--builtin_ntp_servers=127.3.75.20:41355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.75.62:37753
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40161/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790384372073861-3372-0/testchainca.cert with env {}
W20260926 00:59:57.384311 8478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 00:59:57.384502 8478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 00:59:57.384523 8478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 00:59:57.385560 8478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 00:59:57.385592 8478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 00:59:57.385604 8478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 00:59:57.385617 8478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 00:59:57.385628 8478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 00:59:57.385639 8478 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 00:59:57.385651 8478 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40161/jwks.json
I20260926 00:59:57.387115 8478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:41355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790384372073861-3372-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790384372073861-3372-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.75.62:37753
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:37753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40161/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.75.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790384372073861-3372-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790384372073861-3372-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 00:59:57.387288 8478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 00:59:57.387436 8478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 00:59:57.392124 8486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:57.392162 8484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:57.392143 8483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 00:59:57.392195 8478 server_base.cc:1034] running on GCE node
I20260926 00:59:57.393472 8478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 00:59:57.393735 8478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 00:59:57.394914 8478 hybrid_clock.cc:648] HybridClock initialized: now 1790384397394899 us; error 42 us; skew 500 ppm
I20260926 00:59:57.396032 8478 webserver.cc:460] Webserver started at http://127.3.75.62:41481/ using document root <none> and password file <none>
I20260926 00:59:57.396306 8478 fs_manager.cc:359] Metadata directory not provided
I20260926 00:59:57.396376 8478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 00:59:57.396514 8478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 00:59:57.397526 8478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790384372073861-3372-0/minicluster-data/master-0/data/instance:
uuid: "79dfe850ed104d3686209f7d53747fa3"
format_stamp: "Formatted at 2026-09-26 00:59:57 on dist-test-slave-7v10"
server_key: "1a600a475625db8ff6d5e32d75c5fc59"
server_key_iv: "4183e91cd88ae301f0b3276164b3719c"
server_key_version: "encryptionkey@0"
I20260926 00:59:57.397871 8478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790384372073861-3372-0/minicluster-data/master-0/wal/instance:
uuid: "79dfe850ed104d3686209f7d53747fa3"
format_stamp: "Formatted at 2026-09-26 00:59:57 on dist-test-slave-7v10"
server_key: "1a600a475625db8ff6d5e32d75c5fc59"
server_key_iv: "4183e91cd88ae301f0b3276164b3719c"
server_key_version: "encryptionkey@0"
I20260926 00:59:57.399369 8478 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 00:59:57.400069 8492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:57.400233 8478 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 00:59:57.400324 8478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790384372073861-3372-0/minicluster-data/master-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790384372073861-3372-0/minicluster-data/master-0/wal
uuid: "79dfe850ed104d3686209f7d53747fa3"
format_stamp: "Formatted at 2026-09-26 00:59:57 on dist-test-slave-7v10"
server_key: "1a600a475625db8ff6d5e32d75c5fc59"
server_key_iv: "4183e91cd88ae301f0b3276164b3719c"
server_key_version: "encryptionkey@0"
I20260926 00:59:57.400413 8478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790384372073861-3372-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790384372073861-3372-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790384372073861-3372-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 00:59:57.416488 8478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 00:59:57.416765 8478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 00:59:57.416914 8478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 00:59:57.420674 8478 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.62:37753
I20260926 00:59:57.420702 8544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.62:37753 every 8 connection(s)
I20260926 00:59:57.421048 8478 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
I20260926 00:59:57.421988 8545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 00:59:57.424306 8545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79dfe850ed104d3686209f7d53747fa3: Bootstrap starting.
I20260926 00:59:57.424939 8545 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79dfe850ed104d3686209f7d53747fa3: Neither blocks nor log segments found. Creating new log.
I20260926 00:59:57.425249 8545 log.cc:824] T 00000000000000000000000000000000 P 79dfe850ed104d3686209f7d53747fa3: Log is configured to *not* fsync() on all Append() calls
I20260926 00:59:57.425972 8545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79dfe850ed104d3686209f7d53747fa3: No bootstrap required, opened a new log
I20260926 00:59:57.427611 8545 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79dfe850ed104d3686209f7d53747fa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79dfe850ed104d3686209f7d53747fa3" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 37753 } }
I20260926 00:59:57.427742 8545 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79dfe850ed104d3686209f7d53747fa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 00:59:57.427788 8545 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79dfe850ed104d3686209f7d53747fa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79dfe850ed104d3686209f7d53747fa3, State: Initialized, Role: FOLLOWER
I20260926 00:59:57.427955 8545 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79dfe850ed104d3686209f7d53747fa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79dfe850ed104d3686209f7d53747fa3" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 37753 } }
I20260926 00:59:57.428041 8545 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79dfe850ed104d3686209f7d53747fa3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 00:59:57.428109 8545 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79dfe850ed104d3686209f7d53747fa3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 00:59:57.428169 8545 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79dfe850ed104d3686209f7d53747fa3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 00:59:57.428660 8545 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79dfe850ed104d3686209f7d53747fa3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79dfe850ed104d3686209f7d53747fa3" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 37753 } }
I20260926 00:59:57.428771 8545 leader_election.cc:302] T 00000000000000000000000000000000 P 79dfe850ed104d3686209f7d53747fa3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79dfe850ed104d3686209f7d53747fa3; no voters:
I20260926 00:59:57.428957 8545 leader_election.cc:288] T 00000000000000000000000000000000 P 79dfe850ed104d3686209f7d53747fa3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 00:59:57.429016 8548 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79dfe850ed104d3686209f7d53747fa3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 00:59:57.429175 8548 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79dfe850ed104d3686209f7d53747fa3 [term 1 LEADER]: Becoming Leader. State: Replica: 79dfe850ed104d3686209f7d53747fa3, State: Running, Role: LEADER
I20260926 00:59:57.429392 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 8478
I20260926 00:59:57.429492 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790384372073861-3372-0/minicluster-data/master-0/wal/instance
I20260926 00:59:57.429673 3372 external_mini_cluster.cc:1428] Setting key 304a206d7c0ff1a5dcffc9075fefd673
I20260926 00:59:57.429195 8545 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79dfe850ed104d3686209f7d53747fa3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 00:59:57.429943 8548 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79dfe850ed104d3686209f7d53747fa3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79dfe850ed104d3686209f7d53747fa3" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 37753 } }
I20260926 00:59:57.430745 8550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79dfe850ed104d3686209f7d53747fa3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79dfe850ed104d3686209f7d53747fa3. Latest consensus state: current_term: 1 leader_uuid: "79dfe850ed104d3686209f7d53747fa3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79dfe850ed104d3686209f7d53747fa3" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 37753 } } }
I20260926 00:59:57.430831 8550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79dfe850ed104d3686209f7d53747fa3 [sys.catalog]: This master's current role is: LEADER
I20260926 00:59:57.431130 8559 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 00:59:57.431141 8549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79dfe850ed104d3686209f7d53747fa3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79dfe850ed104d3686209f7d53747fa3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79dfe850ed104d3686209f7d53747fa3" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 37753 } } }
I20260926 00:59:57.431245 8549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79dfe850ed104d3686209f7d53747fa3 [sys.catalog]: This master's current role is: LEADER
I20260926 00:59:57.431761 8559 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 00:59:57.433437 8559 catalog_manager.cc:1347] Generated new cluster ID: 446c6d592fc3446b95e3677e861c6914
I20260926 00:59:57.433490 8559 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 00:59:57.436094 8552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:57.430140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60552 (local address 127.3.75.62:37753)
0926 00:59:57.430641 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:57.430651 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:57.430687 (+ 36us) server_negotiation.cc:407] Connection header received
0926 00:59:57.430807 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:57.430814 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:57.430919 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:57.431061 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:57.431430 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:57.432811 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:57.433954 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:57.434525 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:57.434935 (+ 410us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:57.435322 (+ 387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:57.435326 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:57.435326 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 00:59:57.435521 (+ 195us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:57.435556 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 00:59:57.435569 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:57.435759 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:57.435786 (+ 27us) server_negotiation.cc:299] Negotiation successful
0926 00:59:57.435894 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":60,"threads_started":1}
I20260926 00:59:57.437021 3372 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 00:59:57.445207 8559 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 00:59:57.445664 8559 catalog_manager.cc:1504] Loading token signing keys...
I20260926 00:59:57.448575 8559 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79dfe850ed104d3686209f7d53747fa3: Generated new TSK 0
I20260926 00:59:57.448724 8559 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 00:59:57.481657 8575 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260926 00:59:57.481807 8575 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 00:59:57.481933 8552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:57.459593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60568 (local address 127.3.75.62:37753)
0926 00:59:57.459630 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:57.459632 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:57.459754 (+ 122us) server_negotiation.cc:407] Connection header received
0926 00:59:57.459851 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:57.459851 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:57.459862 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:57.459916 (+ 54us) server_negotiation.cc:226] Negotiated authn=JWT
0926 00:59:57.460078 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:57.460536 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:57.461696 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:57.461887 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:57.461931 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:57.462159 (+ 228us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 00:59:57.481659 (+ 19500us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40161/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 00:59:57.481848 (+ 189us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60568: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40161/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 00:59:57.481994 8552 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60568: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40161/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260926 00:59:57.482237 8574 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 00:59:57.459501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:37753 (local address 127.0.0.1:60568)
0926 00:59:57.459684 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:57.459695 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:57.459762 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:57.459941 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:57.459942 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:57.459944 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0926 00:59:57.460013 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:57.460016 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:57.460624 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:57.460625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:57.461609 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:57.461613 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:57.461955 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:57.461958 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:57.462031 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:57.462054 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 00:59:57.481798 (+ 19744us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40161/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 00:59:57.481922 (+ 124us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.75.62:37753: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40161/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":86,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.3.75.62:37753: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40161/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260926 00:59:57.482528 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 8478
2026-09-26T00:59:57Z chronyd exiting
I20260926 00:59:57.518975 3372 test_util.cc:175] -----------------------------------------------
I20260926 00:59:57.519047 3372 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790384372073861-3372-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (223 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260926 00:59:57.519780 3372 mini_oidc.cc:152] Starting JWKS server
I20260926 00:59:57.520411 3372 webserver.cc:460] Webserver started at https://127.0.0.1:35607/ using document root <none> and password file <none>
I20260926 00:59:57.520448 3372 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 00:59:57.520578 3372 webserver.cc:460] Webserver started at http://127.0.0.1:43829/ using document root <none> and password file <none>
2026-09-26T00:59:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T00:59:57Z Disabled control of system clock
I20260926 00:59:57.527954 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790384372073861-3372-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790384372073861-3372-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790384372073861-3372-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:38805
--webserver_interface=127.3.75.62
--webserver_port=0
--builtin_ntp_servers=127.3.75.20:38539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.75.62:38805
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35607/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790384372073861-3372-0/testchainca.cert with env {}
W20260926 00:59:57.608022 8581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 00:59:57.608253 8581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 00:59:57.608291 8581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 00:59:57.609414 8581 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 00:59:57.609483 8581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 00:59:57.609510 8581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 00:59:57.609546 8581 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 00:59:57.609573 8581 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 00:59:57.609608 8581 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 00:59:57.609638 8581 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35607/jwks.json
I20260926 00:59:57.611104 8581 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:38539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790384372073861-3372-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790384372073861-3372-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.75.62:38805
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:38805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35607/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.75.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790384372073861-3372-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790384372073861-3372-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 00:59:57.611387 8581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 00:59:57.611661 8581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 00:59:57.616430 8589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:57.616422 8586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:57.616423 8587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 00:59:57.616787 8581 server_base.cc:1034] running on GCE node
I20260926 00:59:57.616969 8581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 00:59:57.617247 8581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 00:59:57.618404 8581 hybrid_clock.cc:648] HybridClock initialized: now 1790384397618400 us; error 38 us; skew 500 ppm
I20260926 00:59:57.619652 8581 webserver.cc:460] Webserver started at http://127.3.75.62:41937/ using document root <none> and password file <none>
I20260926 00:59:57.619899 8581 fs_manager.cc:359] Metadata directory not provided
I20260926 00:59:57.619968 8581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 00:59:57.620128 8581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 00:59:57.621160 8581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790384372073861-3372-0/minicluster-data/master-0/data/instance:
uuid: "5911be3badac4f1998fd0698a87ed56b"
format_stamp: "Formatted at 2026-09-26 00:59:57 on dist-test-slave-7v10"
server_key: "0d7eec3181a964a4586a5171812cb1ff"
server_key_iv: "d6dd1ca65324e4985a7c0f501deb1784"
server_key_version: "encryptionkey@0"
I20260926 00:59:57.621531 8581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790384372073861-3372-0/minicluster-data/master-0/wal/instance:
uuid: "5911be3badac4f1998fd0698a87ed56b"
format_stamp: "Formatted at 2026-09-26 00:59:57 on dist-test-slave-7v10"
server_key: "0d7eec3181a964a4586a5171812cb1ff"
server_key_iv: "d6dd1ca65324e4985a7c0f501deb1784"
server_key_version: "encryptionkey@0"
I20260926 00:59:57.622797 8581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 00:59:57.623569 8595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:57.623747 8581 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 00:59:57.623842 8581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790384372073861-3372-0/minicluster-data/master-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790384372073861-3372-0/minicluster-data/master-0/wal
uuid: "5911be3badac4f1998fd0698a87ed56b"
format_stamp: "Formatted at 2026-09-26 00:59:57 on dist-test-slave-7v10"
server_key: "0d7eec3181a964a4586a5171812cb1ff"
server_key_iv: "d6dd1ca65324e4985a7c0f501deb1784"
server_key_version: "encryptionkey@0"
I20260926 00:59:57.623929 8581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790384372073861-3372-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790384372073861-3372-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790384372073861-3372-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 00:59:57.650672 8581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 00:59:57.651063 8581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 00:59:57.651228 8581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 00:59:57.654860 8581 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.62:38805
I20260926 00:59:57.654996 8647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.62:38805 every 8 connection(s)
I20260926 00:59:57.655298 8581 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
I20260926 00:59:57.656203 8648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 00:59:57.658403 8648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5911be3badac4f1998fd0698a87ed56b: Bootstrap starting.
I20260926 00:59:57.659032 8648 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5911be3badac4f1998fd0698a87ed56b: Neither blocks nor log segments found. Creating new log.
I20260926 00:59:57.659318 8648 log.cc:824] T 00000000000000000000000000000000 P 5911be3badac4f1998fd0698a87ed56b: Log is configured to *not* fsync() on all Append() calls
I20260926 00:59:57.659973 8648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5911be3badac4f1998fd0698a87ed56b: No bootstrap required, opened a new log
I20260926 00:59:57.661499 8648 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5911be3badac4f1998fd0698a87ed56b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5911be3badac4f1998fd0698a87ed56b" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 38805 } }
I20260926 00:59:57.661628 8648 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5911be3badac4f1998fd0698a87ed56b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 00:59:57.661685 8648 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5911be3badac4f1998fd0698a87ed56b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5911be3badac4f1998fd0698a87ed56b, State: Initialized, Role: FOLLOWER
I20260926 00:59:57.661859 8648 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5911be3badac4f1998fd0698a87ed56b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5911be3badac4f1998fd0698a87ed56b" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 38805 } }
I20260926 00:59:57.661921 8648 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5911be3badac4f1998fd0698a87ed56b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 00:59:57.661942 8648 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5911be3badac4f1998fd0698a87ed56b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 00:59:57.662005 8648 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5911be3badac4f1998fd0698a87ed56b [term 0 FOLLOWER]: Advancing to term 1
I20260926 00:59:57.662391 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 8581
I20260926 00:59:57.662508 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790384372073861-3372-0/minicluster-data/master-0/wal/instance
I20260926 00:59:57.662705 3372 external_mini_cluster.cc:1428] Setting key 2754c61bab834e8e72407b5bab069bd5
I20260926 00:59:57.662753 8648 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5911be3badac4f1998fd0698a87ed56b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5911be3badac4f1998fd0698a87ed56b" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 38805 } }
I20260926 00:59:57.662863 8648 leader_election.cc:302] T 00000000000000000000000000000000 P 5911be3badac4f1998fd0698a87ed56b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5911be3badac4f1998fd0698a87ed56b; no voters:
I20260926 00:59:57.663040 8648 leader_election.cc:288] T 00000000000000000000000000000000 P 5911be3badac4f1998fd0698a87ed56b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 00:59:57.663062 8651 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5911be3badac4f1998fd0698a87ed56b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 00:59:57.663282 8648 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5911be3badac4f1998fd0698a87ed56b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 00:59:57.663288 8651 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5911be3badac4f1998fd0698a87ed56b [term 1 LEADER]: Becoming Leader. State: Replica: 5911be3badac4f1998fd0698a87ed56b, State: Running, Role: LEADER
I20260926 00:59:57.663477 8651 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5911be3badac4f1998fd0698a87ed56b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5911be3badac4f1998fd0698a87ed56b" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 38805 } }
I20260926 00:59:57.664055 8654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5911be3badac4f1998fd0698a87ed56b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5911be3badac4f1998fd0698a87ed56b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5911be3badac4f1998fd0698a87ed56b" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 38805 } } }
I20260926 00:59:57.664149 8654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5911be3badac4f1998fd0698a87ed56b [sys.catalog]: This master's current role is: LEADER
I20260926 00:59:57.664372 8661 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 00:59:57.664410 8652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5911be3badac4f1998fd0698a87ed56b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5911be3badac4f1998fd0698a87ed56b. Latest consensus state: current_term: 1 leader_uuid: "5911be3badac4f1998fd0698a87ed56b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5911be3badac4f1998fd0698a87ed56b" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 38805 } } }
I20260926 00:59:57.664695 8652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5911be3badac4f1998fd0698a87ed56b [sys.catalog]: This master's current role is: LEADER
I20260926 00:59:57.665045 8661 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 00:59:57.666976 8661 catalog_manager.cc:1347] Generated new cluster ID: 9a39c66f9803464387a573c8a4b0e091
I20260926 00:59:57.667029 8661 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 00:59:57.669039 8655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:57.663175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43738 (local address 127.3.75.62:38805)
0926 00:59:57.663787 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:57.663797 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:57.663831 (+ 34us) server_negotiation.cc:407] Connection header received
0926 00:59:57.663938 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:57.663945 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:57.664005 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:57.664140 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:57.664538 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:57.665857 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:57.666966 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:57.667551 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:57.667665 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:57.668361 (+ 696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:57.668365 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:57.668365 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 00:59:57.668568 (+ 203us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:57.668606 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 00:59:57.668619 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:57.668681 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:57.668778 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 00:59:57.668847 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":95,"threads_started":1}
I20260926 00:59:57.669786 3372 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 00:59:57.685945 8661 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 00:59:57.686528 8661 catalog_manager.cc:1504] Loading token signing keys...
I20260926 00:59:57.693171 8661 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5911be3badac4f1998fd0698a87ed56b: Generated new TSK 0
I20260926 00:59:57.693384 8661 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 00:59:57.715550 8678 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260926 00:59:57.715642 8678 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 00:59:57.716048 8655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:57.701592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43750 (local address 127.3.75.62:38805)
0926 00:59:57.701657 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:57.701659 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:57.701741 (+ 82us) server_negotiation.cc:407] Connection header received
0926 00:59:57.701825 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:57.701826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:57.701837 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:57.701910 (+ 73us) server_negotiation.cc:226] Negotiated authn=JWT
0926 00:59:57.702037 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:57.702539 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:57.703394 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:57.703615 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:57.703692 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:57.703877 (+ 185us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 00:59:57.715794 (+ 11917us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35607/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 00:59:57.715973 (+ 179us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43750: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35607/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":39}
W20260926 00:59:57.716111 8655 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43750: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35607/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260926 00:59:57.716113 8677 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 00:59:57.701534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:38805 (local address 127.0.0.1:43750)
0926 00:59:57.701676 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:57.701686 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:57.701754 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:57.701911 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:57.701912 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:57.701914 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0926 00:59:57.701970 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:57.701973 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:57.702625 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:57.702627 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:57.703305 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:57.703308 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:57.703696 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:57.703700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:57.703772 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:57.703777 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 00:59:57.715917 (+ 12140us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35607/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 00:59:57.716020 (+ 103us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.75.62:38805: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35607/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":77,"threads_started":1}
I20260926 00:59:57.716610 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 8581
2026-09-26T00:59:57Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (234 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260926 00:59:57.754680 3372 mini_oidc.cc:152] Starting JWKS server
I20260926 00:59:57.755407 3372 webserver.cc:460] Webserver started at https://127.0.0.1:43259/ using document root <none> and password file <none>
I20260926 00:59:57.755448 3372 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 00:59:57.755574 3372 webserver.cc:460] Webserver started at http://127.0.0.1:33857/ using document root <none> and password file <none>
2026-09-26T00:59:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T00:59:57Z Disabled control of system clock
I20260926 00:59:57.763151 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790384372073861-3372-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790384372073861-3372-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790384372073861-3372-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:33309
--webserver_interface=127.3.75.62
--webserver_port=0
--builtin_ntp_servers=127.3.75.20:46011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.75.62:33309
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43259/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 00:59:57.841902 8684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 00:59:57.842088 8684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 00:59:57.842108 8684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 00:59:57.843183 8684 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 00:59:57.843217 8684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 00:59:57.843230 8684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 00:59:57.843242 8684 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 00:59:57.843255 8684 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 00:59:57.843267 8684 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 00:59:57.843279 8684 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43259/jwks.json
I20260926 00:59:57.844540 8684 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:46011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790384372073861-3372-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790384372073861-3372-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.75.62:33309
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:33309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43259/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790384372073861-3372-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 00:59:57.844714 8684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 00:59:57.844930 8684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 00:59:57.849645 8690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:57.849640 8692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:57.849767 8689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 00:59:57.849910 8684 server_base.cc:1034] running on GCE node
I20260926 00:59:57.850131 8684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 00:59:57.850406 8684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 00:59:57.851564 8684 hybrid_clock.cc:648] HybridClock initialized: now 1790384397851563 us; error 39 us; skew 500 ppm
I20260926 00:59:57.852800 8684 webserver.cc:460] Webserver started at http://127.3.75.62:40419/ using document root <none> and password file <none>
I20260926 00:59:57.853051 8684 fs_manager.cc:359] Metadata directory not provided
I20260926 00:59:57.853119 8684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 00:59:57.853258 8684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 00:59:57.854264 8684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790384372073861-3372-0/minicluster-data/master-0/data/instance:
uuid: "73e2ec2f7329404691d6e2858eb39ff7"
format_stamp: "Formatted at 2026-09-26 00:59:57 on dist-test-slave-7v10"
server_key: "096c8edc23e292981a1db1731851e2dd"
server_key_iv: "575622a57a89e30695b3c236ef4bc84c"
server_key_version: "encryptionkey@0"
I20260926 00:59:57.854619 8684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790384372073861-3372-0/minicluster-data/master-0/wal/instance:
uuid: "73e2ec2f7329404691d6e2858eb39ff7"
format_stamp: "Formatted at 2026-09-26 00:59:57 on dist-test-slave-7v10"
server_key: "096c8edc23e292981a1db1731851e2dd"
server_key_iv: "575622a57a89e30695b3c236ef4bc84c"
server_key_version: "encryptionkey@0"
I20260926 00:59:57.855973 8684 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 00:59:57.856688 8698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:57.856839 8684 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 00:59:57.856925 8684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790384372073861-3372-0/minicluster-data/master-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790384372073861-3372-0/minicluster-data/master-0/wal
uuid: "73e2ec2f7329404691d6e2858eb39ff7"
format_stamp: "Formatted at 2026-09-26 00:59:57 on dist-test-slave-7v10"
server_key: "096c8edc23e292981a1db1731851e2dd"
server_key_iv: "575622a57a89e30695b3c236ef4bc84c"
server_key_version: "encryptionkey@0"
I20260926 00:59:57.856976 8684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790384372073861-3372-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790384372073861-3372-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790384372073861-3372-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 00:59:57.873627 8684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 00:59:57.873880 8684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 00:59:57.874007 8684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 00:59:57.877777 8684 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.62:33309
I20260926 00:59:57.877867 8750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.62:33309 every 8 connection(s)
I20260926 00:59:57.878149 8684 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
I20260926 00:59:57.878777 8751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 00:59:57.881228 8751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73e2ec2f7329404691d6e2858eb39ff7: Bootstrap starting.
I20260926 00:59:57.881728 8751 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73e2ec2f7329404691d6e2858eb39ff7: Neither blocks nor log segments found. Creating new log.
I20260926 00:59:57.881963 8751 log.cc:824] T 00000000000000000000000000000000 P 73e2ec2f7329404691d6e2858eb39ff7: Log is configured to *not* fsync() on all Append() calls
I20260926 00:59:57.882547 8751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73e2ec2f7329404691d6e2858eb39ff7: No bootstrap required, opened a new log
I20260926 00:59:57.884068 8751 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73e2ec2f7329404691d6e2858eb39ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73e2ec2f7329404691d6e2858eb39ff7" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 33309 } }
I20260926 00:59:57.884196 8751 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73e2ec2f7329404691d6e2858eb39ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 00:59:57.884219 8751 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73e2ec2f7329404691d6e2858eb39ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73e2ec2f7329404691d6e2858eb39ff7, State: Initialized, Role: FOLLOWER
I20260926 00:59:57.884327 8751 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73e2ec2f7329404691d6e2858eb39ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73e2ec2f7329404691d6e2858eb39ff7" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 33309 } }
I20260926 00:59:57.884377 8751 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73e2ec2f7329404691d6e2858eb39ff7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 00:59:57.884397 8751 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73e2ec2f7329404691d6e2858eb39ff7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 00:59:57.884421 8751 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73e2ec2f7329404691d6e2858eb39ff7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 00:59:57.884874 8751 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73e2ec2f7329404691d6e2858eb39ff7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73e2ec2f7329404691d6e2858eb39ff7" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 33309 } }
I20260926 00:59:57.884953 8751 leader_election.cc:302] T 00000000000000000000000000000000 P 73e2ec2f7329404691d6e2858eb39ff7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73e2ec2f7329404691d6e2858eb39ff7; no voters:
I20260926 00:59:57.885069 8751 leader_election.cc:288] T 00000000000000000000000000000000 P 73e2ec2f7329404691d6e2858eb39ff7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 00:59:57.885124 8754 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73e2ec2f7329404691d6e2858eb39ff7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 00:59:57.885303 8754 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73e2ec2f7329404691d6e2858eb39ff7 [term 1 LEADER]: Becoming Leader. State: Replica: 73e2ec2f7329404691d6e2858eb39ff7, State: Running, Role: LEADER
I20260926 00:59:57.885317 8751 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73e2ec2f7329404691d6e2858eb39ff7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 00:59:57.885430 8754 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73e2ec2f7329404691d6e2858eb39ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73e2ec2f7329404691d6e2858eb39ff7" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 33309 } }
I20260926 00:59:57.885793 8756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73e2ec2f7329404691d6e2858eb39ff7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73e2ec2f7329404691d6e2858eb39ff7. Latest consensus state: current_term: 1 leader_uuid: "73e2ec2f7329404691d6e2858eb39ff7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73e2ec2f7329404691d6e2858eb39ff7" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 33309 } } }
I20260926 00:59:57.885918 8755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73e2ec2f7329404691d6e2858eb39ff7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "73e2ec2f7329404691d6e2858eb39ff7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73e2ec2f7329404691d6e2858eb39ff7" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 33309 } } }
I20260926 00:59:57.885941 8756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73e2ec2f7329404691d6e2858eb39ff7 [sys.catalog]: This master's current role is: LEADER
I20260926 00:59:57.886008 8755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73e2ec2f7329404691d6e2858eb39ff7 [sys.catalog]: This master's current role is: LEADER
I20260926 00:59:57.886410 8763 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 00:59:57.886824 8763 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 00:59:57.887156 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 8684
I20260926 00:59:57.887277 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790384372073861-3372-0/minicluster-data/master-0/wal/instance
I20260926 00:59:57.887444 3372 external_mini_cluster.cc:1428] Setting key 2346a4f609c8b8b230379b59327bc8f7
I20260926 00:59:57.888465 8763 catalog_manager.cc:1347] Generated new cluster ID: 5e3d9bbdc92b4aea8a0ddbf0029b098f
I20260926 00:59:57.888516 8763 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 00:59:57.892318 8763 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 00:59:57.892789 8763 catalog_manager.cc:1504] Loading token signing keys...
I20260926 00:59:57.895890 8773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:57.887831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48768 (local address 127.3.75.62:33309)
0926 00:59:57.890664 (+ 2833us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:57.890672 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:57.890704 (+ 32us) server_negotiation.cc:407] Connection header received
0926 00:59:57.890815 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:57.890833 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:57.890945 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:57.891095 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:57.891320 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:57.892693 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:57.893752 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:57.894362 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:57.894487 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:57.895177 (+ 690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:57.895181 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:57.895181 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 00:59:57.895390 (+ 209us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:57.895431 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 00:59:57.895445 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:57.895510 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:57.895584 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 00:59:57.895670 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2760,"thread_start_us":72,"threads_started":1}
I20260926 00:59:57.896741 3372 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 00:59:57.897150 8763 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73e2ec2f7329404691d6e2858eb39ff7: Generated new TSK 0
I20260926 00:59:57.897290 8763 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 00:59:57.905049 8773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:57.901523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48772 (local address 127.3.75.62:33309)
0926 00:59:57.901582 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:57.901584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:57.901745 (+ 161us) server_negotiation.cc:407] Connection header received
0926 00:59:57.901794 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:57.901794 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:57.901805 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:57.901890 (+ 85us) server_negotiation.cc:226] Negotiated authn=JWT
0926 00:59:57.901976 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:57.902702 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:57.903493 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:57.903788 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:57.903870 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:57.903982 (+ 112us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 00:59:57.904856 (+ 874us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43259/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 00:59:57.904978 (+ 122us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48772: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43259/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":38}
W20260926 00:59:57.905097 8773 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48772: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43259/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260926 00:59:57.906934 8780 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 00:59:57.901558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:33309 (local address 127.0.0.1:48772)
0926 00:59:57.901675 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:57.901685 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:57.901744 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:57.901886 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:57.901887 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:57.901888 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0926 00:59:57.901928 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:57.901929 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:57.902789 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:57.902790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:57.903420 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:57.903422 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:57.903868 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:57.903871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:57.903909 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:57.903912 (+ 3us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 00:59:57.904957 (+ 1045us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43259/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 00:59:57.906777 (+ 1820us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.75.62:33309: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43259/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":96,"mutex_wait_us":1643,"thread_start_us":78,"threads_started":1}
I20260926 00:59:57.907155 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 8684
E20260926 00:59:57.916134 8781 webserver.cc:533] Webserver: sslize failed:
E20260926 00:59:57.916215 8781 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-26T00:59:57Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (189 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-26T00:59:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T00:59:57Z Disabled control of system clock
I20260926 00:59:57.951699 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790384372073861-3372-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790384372073861-3372-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790384372073861-3372-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:35995
--webserver_interface=127.3.75.62
--webserver_port=0
--builtin_ntp_servers=127.3.75.20:45853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.75.62:35995
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790384372073861-3372-0/testchainca.cert with env {}
W20260926 00:59:58.031201 8785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 00:59:58.031440 8785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 00:59:58.031479 8785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 00:59:58.033142 8785 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 00:59:58.033186 8785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 00:59:58.033200 8785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 00:59:58.033213 8785 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 00:59:58.033226 8785 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 00:59:58.033238 8785 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 00:59:58.033250 8785 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260926 00:59:58.034679 8785 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:45853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790384372073861-3372-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790384372073861-3372-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.75.62:35995
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:35995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.75.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790384372073861-3372-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790384372073861-3372-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 00:59:58.034911 8785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 00:59:58.035099 8785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 00:59:58.039909 8791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:58.039927 8790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:58.039884 8793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 00:59:58.040170 8785 server_base.cc:1034] running on GCE node
I20260926 00:59:58.040428 8785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 00:59:58.040664 8785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 00:59:58.041808 8785 hybrid_clock.cc:648] HybridClock initialized: now 1790384398041795 us; error 28 us; skew 500 ppm
I20260926 00:59:58.043033 8785 webserver.cc:460] Webserver started at http://127.3.75.62:36541/ using document root <none> and password file <none>
I20260926 00:59:58.043262 8785 fs_manager.cc:359] Metadata directory not provided
I20260926 00:59:58.043322 8785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 00:59:58.043479 8785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 00:59:58.044507 8785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790384372073861-3372-0/minicluster-data/master-0/data/instance:
uuid: "40c69830340f4faeb62b2cdeb0a05974"
format_stamp: "Formatted at 2026-09-26 00:59:58 on dist-test-slave-7v10"
server_key: "854d1df9e46151ecd42721b949781228"
server_key_iv: "abd207b7f273ce3a5b329962503b169f"
server_key_version: "encryptionkey@0"
I20260926 00:59:58.044863 8785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790384372073861-3372-0/minicluster-data/master-0/wal/instance:
uuid: "40c69830340f4faeb62b2cdeb0a05974"
format_stamp: "Formatted at 2026-09-26 00:59:58 on dist-test-slave-7v10"
server_key: "854d1df9e46151ecd42721b949781228"
server_key_iv: "abd207b7f273ce3a5b329962503b169f"
server_key_version: "encryptionkey@0"
I20260926 00:59:58.046203 8785 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 00:59:58.046937 8799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:58.047115 8785 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 00:59:58.047214 8785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790384372073861-3372-0/minicluster-data/master-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790384372073861-3372-0/minicluster-data/master-0/wal
uuid: "40c69830340f4faeb62b2cdeb0a05974"
format_stamp: "Formatted at 2026-09-26 00:59:58 on dist-test-slave-7v10"
server_key: "854d1df9e46151ecd42721b949781228"
server_key_iv: "abd207b7f273ce3a5b329962503b169f"
server_key_version: "encryptionkey@0"
I20260926 00:59:58.047297 8785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790384372073861-3372-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790384372073861-3372-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790384372073861-3372-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 00:59:58.056646 8785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 00:59:58.056934 8785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 00:59:58.057080 8785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 00:59:58.060886 8785 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.62:35995
I20260926 00:59:58.060915 8851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.62:35995 every 8 connection(s)
I20260926 00:59:58.061265 8785 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
I20260926 00:59:58.062150 8852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 00:59:58.064359 8852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40c69830340f4faeb62b2cdeb0a05974: Bootstrap starting.
I20260926 00:59:58.064886 8852 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40c69830340f4faeb62b2cdeb0a05974: Neither blocks nor log segments found. Creating new log.
I20260926 00:59:58.065124 8852 log.cc:824] T 00000000000000000000000000000000 P 40c69830340f4faeb62b2cdeb0a05974: Log is configured to *not* fsync() on all Append() calls
I20260926 00:59:58.065717 8852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40c69830340f4faeb62b2cdeb0a05974: No bootstrap required, opened a new log
I20260926 00:59:58.065828 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 8785
I20260926 00:59:58.065958 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790384372073861-3372-0/minicluster-data/master-0/wal/instance
I20260926 00:59:58.066148 3372 external_mini_cluster.cc:1428] Setting key af6737d3ce4b7bc6fe0d0b9363523802
I20260926 00:59:58.067729 8852 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40c69830340f4faeb62b2cdeb0a05974 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40c69830340f4faeb62b2cdeb0a05974" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 35995 } }
I20260926 00:59:58.067898 8852 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40c69830340f4faeb62b2cdeb0a05974 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 00:59:58.067950 8852 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40c69830340f4faeb62b2cdeb0a05974 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c69830340f4faeb62b2cdeb0a05974, State: Initialized, Role: FOLLOWER
I20260926 00:59:58.068136 8852 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40c69830340f4faeb62b2cdeb0a05974 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40c69830340f4faeb62b2cdeb0a05974" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 35995 } }
I20260926 00:59:58.068225 8852 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40c69830340f4faeb62b2cdeb0a05974 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 00:59:58.068259 8852 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40c69830340f4faeb62b2cdeb0a05974 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 00:59:58.068312 8852 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40c69830340f4faeb62b2cdeb0a05974 [term 0 FOLLOWER]: Advancing to term 1
I20260926 00:59:58.068849 8852 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40c69830340f4faeb62b2cdeb0a05974 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40c69830340f4faeb62b2cdeb0a05974" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 35995 } }
I20260926 00:59:58.068964 8852 leader_election.cc:302] T 00000000000000000000000000000000 P 40c69830340f4faeb62b2cdeb0a05974 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40c69830340f4faeb62b2cdeb0a05974; no voters:
I20260926 00:59:58.069113 8852 leader_election.cc:288] T 00000000000000000000000000000000 P 40c69830340f4faeb62b2cdeb0a05974 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 00:59:58.069156 8857 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40c69830340f4faeb62b2cdeb0a05974 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 00:59:58.069315 8857 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40c69830340f4faeb62b2cdeb0a05974 [term 1 LEADER]: Becoming Leader. State: Replica: 40c69830340f4faeb62b2cdeb0a05974, State: Running, Role: LEADER
I20260926 00:59:58.069360 8852 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40c69830340f4faeb62b2cdeb0a05974 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 00:59:58.069444 8857 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40c69830340f4faeb62b2cdeb0a05974 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40c69830340f4faeb62b2cdeb0a05974" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 35995 } }
I20260926 00:59:58.069741 8857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40c69830340f4faeb62b2cdeb0a05974 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40c69830340f4faeb62b2cdeb0a05974. Latest consensus state: current_term: 1 leader_uuid: "40c69830340f4faeb62b2cdeb0a05974" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40c69830340f4faeb62b2cdeb0a05974" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 35995 } } }
I20260926 00:59:58.069832 8857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40c69830340f4faeb62b2cdeb0a05974 [sys.catalog]: This master's current role is: LEADER
I20260926 00:59:58.069752 8858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40c69830340f4faeb62b2cdeb0a05974 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40c69830340f4faeb62b2cdeb0a05974" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40c69830340f4faeb62b2cdeb0a05974" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 35995 } } }
I20260926 00:59:58.069972 8858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40c69830340f4faeb62b2cdeb0a05974 [sys.catalog]: This master's current role is: LEADER
I20260926 00:59:58.071027 8863 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 00:59:58.071664 8863 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 00:59:58.073096 8856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:58.066664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38542 (local address 127.3.75.62:35995)
0926 00:59:58.066849 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:58.066857 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:58.066921 (+ 64us) server_negotiation.cc:407] Connection header received
0926 00:59:58.067714 (+ 793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:58.067721 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:58.067775 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:58.067888 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:58.068169 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:58.069720 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:58.070773 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:58.071451 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:58.071569 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:58.072282 (+ 713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:58.072286 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:58.072286 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 00:59:58.072477 (+ 191us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:58.072508 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 00:59:58.072518 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:58.072597 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:58.072674 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 00:59:58.072849 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":75,"threads_started":1}
I20260926 00:59:58.073402 8863 catalog_manager.cc:1347] Generated new cluster ID: 6b6d566530554ab7b4c8818de4a3f17a
I20260926 00:59:58.073451 8863 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 00:59:58.073935 3372 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 00:59:58.087954 8863 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 00:59:58.088490 8863 catalog_manager.cc:1504] Loading token signing keys...
I20260926 00:59:58.093605 8863 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40c69830340f4faeb62b2cdeb0a05974: Generated new TSK 0
I20260926 00:59:58.093798 8863 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 00:59:58.268249 8856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:58.102361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38554 (local address 127.3.75.62:35995)
0926 00:59:58.102427 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:58.102429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:58.102555 (+ 126us) server_negotiation.cc:407] Connection header received
0926 00:59:58.102623 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:58.102623 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:58.102633 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:58.102691 (+ 58us) server_negotiation.cc:226] Negotiated authn=JWT
0926 00:59:58.102840 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:58.103393 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:58.104534 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:58.104713 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:58.104769 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:58.104960 (+ 191us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 00:59:58.268019 (+163059us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 00:59:58.268188 (+ 169us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38554: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":39}
W20260926 00:59:58.268345 8856 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38554: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260926 00:59:58.268577 8881 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 00:59:58.102296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:35995 (local address 127.0.0.1:38554)
0926 00:59:58.102477 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:58.102489 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:58.102555 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:58.102710 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:58.102711 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:58.102713 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0926 00:59:58.102771 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:58.102774 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:58.103476 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:58.103477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:58.104449 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:58.104452 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:58.104788 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:58.104792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:58.104861 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:58.104867 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 00:59:58.268208 (+163341us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 00:59:58.268351 (+ 143us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.75.62:35995: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":116,"threads_started":1}
I20260926 00:59:58.268967 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 8785
2026-09-26T00:59:58Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (342 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (86 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (85 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 00:59:58 dist-test-slave-7v10 krb5kdc[8893](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 00:59:58 dist-test-slave-7v10 krb5kdc[8893](info): set up 2 sockets
Sep 26 00:59:58 dist-test-slave-7v10 krb5kdc[8893](info): commencing operation
krb5kdc: starting...
W20260926 00:59:59.478124 3372 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Sep 26 00:59:59 dist-test-slave-7v10 krb5kdc[8893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T00:59:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T00:59:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.75.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab.
Entry for principal kudu/127.3.75.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab.
I20260926 00:59:59.590523 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:45843
--webserver_interface=127.3.75.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:34621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.75.62:45843
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 00:59:59.673292 8906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 00:59:59.673502 8906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 00:59:59.673523 8906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 00:59:59.674685 8906 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 00:59:59.674724 8906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 00:59:59.674739 8906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 00:59:59.674751 8906 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 00:59:59.674764 8906 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 00:59:59.676136 8906 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:34621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.75.62:45843
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:45843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 00:59:59.676323 8906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 00:59:59.676481 8906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 00:59:59.681079 8911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:59.681095 8914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:59.681083 8912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 00:59:59.681385 8906 server_base.cc:1034] running on GCE node
I20260926 00:59:59.681636 8906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 00:59:59.681965 8906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 00:59:59.683125 8906 hybrid_clock.cc:648] HybridClock initialized: now 1790384399683110 us; error 44 us; skew 500 ppm
Sep 26 00:59:59 dist-test-slave-7v10 krb5kdc[8893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384399, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 00:59:59.688377 8906 init.cc:377] Logged in from keytab as kudu/127.3.75.62@KRBTEST.COM (short username kudu)
I20260926 00:59:59.688845 8906 webserver.cc:460] Webserver started at http://127.3.75.62:33769/ using document root <none> and password file <none>
I20260926 00:59:59.689069 8906 fs_manager.cc:359] Metadata directory not provided
I20260926 00:59:59.689136 8906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 00:59:59.689285 8906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 00:59:59.690320 8906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/master-0/data/instance:
uuid: "a2d8878631014b7c8c9a215a1bc81f40"
format_stamp: "Formatted at 2026-09-26 00:59:59 on dist-test-slave-7v10"
server_key: "6dff5bc9ce0d85935b5256f8275507e3"
server_key_iv: "2edf6e66db4473528ffe9c23636d43d0"
server_key_version: "encryptionkey@0"
I20260926 00:59:59.690683 8906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/master-0/wal/instance:
uuid: "a2d8878631014b7c8c9a215a1bc81f40"
format_stamp: "Formatted at 2026-09-26 00:59:59 on dist-test-slave-7v10"
server_key: "6dff5bc9ce0d85935b5256f8275507e3"
server_key_iv: "2edf6e66db4473528ffe9c23636d43d0"
server_key_version: "encryptionkey@0"
I20260926 00:59:59.692054 8906 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 00:59:59.692754 8921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:59.692924 8906 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:59.693022 8906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/master-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/master-0/wal
uuid: "a2d8878631014b7c8c9a215a1bc81f40"
format_stamp: "Formatted at 2026-09-26 00:59:59 on dist-test-slave-7v10"
server_key: "6dff5bc9ce0d85935b5256f8275507e3"
server_key_iv: "2edf6e66db4473528ffe9c23636d43d0"
server_key_version: "encryptionkey@0"
I20260926 00:59:59.693106 8906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 00:59:59.707367 8906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 00:59:59.709021 8906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 00:59:59.709182 8906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 00:59:59.713120 8906 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.62:45843
I20260926 00:59:59.713150 8973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.62:45843 every 8 connection(s)
I20260926 00:59:59.713501 8906 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
I20260926 00:59:59.714329 8974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 00:59:59.714815 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 8906
I20260926 00:59:59.714953 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/master-0/wal/instance
I20260926 00:59:59.715127 3372 external_mini_cluster.cc:1428] Setting key 47d571e3e427afb971787cd20d7f2dc9
I20260926 00:59:59.717314 8974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2d8878631014b7c8c9a215a1bc81f40: Bootstrap starting.
I20260926 00:59:59.717907 8974 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2d8878631014b7c8c9a215a1bc81f40: Neither blocks nor log segments found. Creating new log.
I20260926 00:59:59.718200 8974 log.cc:824] T 00000000000000000000000000000000 P a2d8878631014b7c8c9a215a1bc81f40: Log is configured to *not* fsync() on all Append() calls
I20260926 00:59:59.718864 8974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2d8878631014b7c8c9a215a1bc81f40: No bootstrap required, opened a new log
I20260926 00:59:59.720866 8974 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2d8878631014b7c8c9a215a1bc81f40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2d8878631014b7c8c9a215a1bc81f40" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 45843 } }
I20260926 00:59:59.721060 8974 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2d8878631014b7c8c9a215a1bc81f40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 00:59:59.721102 8974 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2d8878631014b7c8c9a215a1bc81f40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2d8878631014b7c8c9a215a1bc81f40, State: Initialized, Role: FOLLOWER
I20260926 00:59:59.721238 8974 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2d8878631014b7c8c9a215a1bc81f40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2d8878631014b7c8c9a215a1bc81f40" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 45843 } }
I20260926 00:59:59.721319 8974 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2d8878631014b7c8c9a215a1bc81f40 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 00:59:59.721351 8974 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2d8878631014b7c8c9a215a1bc81f40 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 00:59:59.721405 8974 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2d8878631014b7c8c9a215a1bc81f40 [term 0 FOLLOWER]: Advancing to term 1
Sep 26 00:59:59 dist-test-slave-7v10 krb5kdc[8893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 00:59:59.722096 8974 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2d8878631014b7c8c9a215a1bc81f40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2d8878631014b7c8c9a215a1bc81f40" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 45843 } }
I20260926 00:59:59.722213 8974 leader_election.cc:302] T 00000000000000000000000000000000 P a2d8878631014b7c8c9a215a1bc81f40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2d8878631014b7c8c9a215a1bc81f40; no voters:
I20260926 00:59:59.722388 8974 leader_election.cc:288] T 00000000000000000000000000000000 P a2d8878631014b7c8c9a215a1bc81f40 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 00:59:59.722432 8979 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2d8878631014b7c8c9a215a1bc81f40 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 00:59:59.722651 8974 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2d8878631014b7c8c9a215a1bc81f40 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 00:59:59.722636 8979 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2d8878631014b7c8c9a215a1bc81f40 [term 1 LEADER]: Becoming Leader. State: Replica: a2d8878631014b7c8c9a215a1bc81f40, State: Running, Role: LEADER
I20260926 00:59:59.723502 8979 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2d8878631014b7c8c9a215a1bc81f40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2d8878631014b7c8c9a215a1bc81f40" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 45843 } }
I20260926 00:59:59.724100 8981 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2d8878631014b7c8c9a215a1bc81f40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2d8878631014b7c8c9a215a1bc81f40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2d8878631014b7c8c9a215a1bc81f40" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 45843 } } }
I20260926 00:59:59.724247 8980 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2d8878631014b7c8c9a215a1bc81f40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2d8878631014b7c8c9a215a1bc81f40. Latest consensus state: current_term: 1 leader_uuid: "a2d8878631014b7c8c9a215a1bc81f40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2d8878631014b7c8c9a215a1bc81f40" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 45843 } } }
I20260926 00:59:59.724496 8981 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2d8878631014b7c8c9a215a1bc81f40 [sys.catalog]: This master's current role is: LEADER
I20260926 00:59:59.724502 8980 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2d8878631014b7c8c9a215a1bc81f40 [sys.catalog]: This master's current role is: LEADER
W20260926 00:59:59.725060 8993 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a2d8878631014b7c8c9a215a1bc81f40: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 00:59:59.725116 8993 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 00:59:59.725207 8995 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 00:59:59.725458 8995 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 00:59:59.726033 8977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:59.715685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33614 (local address 127.3.75.62:45843)
0926 00:59:59.715857 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:59.715865 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:59.715890 (+ 25us) server_negotiation.cc:407] Connection header received
0926 00:59:59.716280 (+ 390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:59.716287 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:59.716346 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:59.716461 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:59.717262 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:59.718233 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:59.719109 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:59.719515 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:59.719642 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:59.722243 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:59.722247 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:59.722247 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 00:59:59.722258 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:59.724379 (+ 2121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:59.724803 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:59.724806 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:59.724807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:59.724932 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:59.725187 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:59.725188 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:59.725189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:59.725422 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:59.725602 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:59.725700 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 00:59:59.725837 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":77,"threads_started":1}
I20260926 00:59:59.726888 8995 catalog_manager.cc:1347] Generated new cluster ID: 21cb426d2115445ebe25f98fecd97ddf
I20260926 00:59:59.726936 8995 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 00:59:59.742549 8995 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 00:59:59.743191 8995 catalog_manager.cc:1504] Loading token signing keys...
I20260926 00:59:59.754585 8995 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2d8878631014b7c8c9a215a1bc81f40: Generated new TSK 0
I20260926 00:59:59.754777 8995 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.75.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab.
Entry for principal kudu/127.3.75.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab.
I20260926 00:59:59.785249 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.1:0
--local_ip_for_outbound_sockets=127.3.75.1
--webserver_interface=127.3.75.1
--webserver_port=0
--tserver_master_addrs=127.3.75.62:45843
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab
--principal=kudu/127.3.75.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:34621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 00:59:59.881174 9000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 00:59:59.881387 9000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 00:59:59.881412 9000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 00:59:59.882486 9000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 00:59:59.882520 9000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 00:59:59.882545 9000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.1
I20260926 00:59:59.883939 9000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:34621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab
--principal=kudu/127.3.75.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.75.1
--webserver_port=0
--tserver_master_addrs=127.3.75.62:45843
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.1
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 00:59:59.884119 9000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 00:59:59.884296 9000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 00:59:59.889173 9008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 00:59:59.889210 9005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 00:59:59.889402 9000 server_base.cc:1034] running on GCE node
W20260926 00:59:59.889170 9006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 00:59:59.889657 9000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 00:59:59.889935 9000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 00:59:59.891084 9000 hybrid_clock.cc:648] HybridClock initialized: now 1790384399891063 us; error 38 us; skew 500 ppm
Sep 26 00:59:59 dist-test-slave-7v10 krb5kdc[8893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384399, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 00:59:59.893431 9000 init.cc:377] Logged in from keytab as kudu/127.3.75.1@KRBTEST.COM (short username kudu)
I20260926 00:59:59.893800 9000 webserver.cc:460] Webserver started at http://127.3.75.1:40635/ using document root <none> and password file <none>
I20260926 00:59:59.894035 9000 fs_manager.cc:359] Metadata directory not provided
I20260926 00:59:59.894079 9000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 00:59:59.894241 9000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 00:59:59.895377 9000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/ts-0/data/instance:
uuid: "29914875df3c4e1eb4dfc87840c2908e"
format_stamp: "Formatted at 2026-09-26 00:59:59 on dist-test-slave-7v10"
server_key: "c43fecb7e0324d2574da376a774f898c"
server_key_iv: "c9d9beff74dc4f2679173174dc462281"
server_key_version: "encryptionkey@0"
I20260926 00:59:59.895776 9000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/ts-0/wal/instance:
uuid: "29914875df3c4e1eb4dfc87840c2908e"
format_stamp: "Formatted at 2026-09-26 00:59:59 on dist-test-slave-7v10"
server_key: "c43fecb7e0324d2574da376a774f898c"
server_key_iv: "c9d9beff74dc4f2679173174dc462281"
server_key_version: "encryptionkey@0"
I20260926 00:59:59.897145 9000 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 00:59:59.897894 9015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:59.898089 9000 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 00:59:59.898180 9000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/ts-0/wal
uuid: "29914875df3c4e1eb4dfc87840c2908e"
format_stamp: "Formatted at 2026-09-26 00:59:59 on dist-test-slave-7v10"
server_key: "c43fecb7e0324d2574da376a774f898c"
server_key_iv: "c9d9beff74dc4f2679173174dc462281"
server_key_version: "encryptionkey@0"
I20260926 00:59:59.898250 9000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 00:59:59.913094 9000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 00:59:59.915411 9000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 00:59:59.915573 9000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 00:59:59.915807 9000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 00:59:59.916116 9000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 00:59:59.916150 9000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:59.916193 9000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 00:59:59.916220 9000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 00:59:59.922055 9000 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.1:36213
I20260926 00:59:59.922137 9128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.1:36213 every 8 connection(s)
I20260926 00:59:59.922459 9000 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
Sep 26 00:59:59 dist-test-slave-7v10 krb5kdc[8893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384399, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.1@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 00:59:59.930605 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 9000
I20260926 00:59:59.930716 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/ts-0/wal/instance
I20260926 00:59:59.930948 3372 external_mini_cluster.cc:1428] Setting key ee15c69dca18670f5ef01d405d65a3a6
I20260926 00:59:59.932137 8977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:59.923375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:59791 (local address 127.3.75.62:45843)
0926 00:59:59.923446 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 00:59:59.923452 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 00:59:59.923641 (+ 189us) server_negotiation.cc:407] Connection header received
0926 00:59:59.924777 (+ 1136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 00:59:59.924779 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 00:59:59.924789 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 00:59:59.924853 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 00:59:59.925423 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:59.925906 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:59.926845 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 00:59:59.927128 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 00:59:59.927181 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:59.929372 (+ 2191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 00:59:59.929373 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 00:59:59.929373 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 00:59:59.929379 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 00:59:59.930554 (+ 1175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:59.931114 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:59.931115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:59.931116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:59.931182 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 00:59:59.931465 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 00:59:59.931466 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 00:59:59.931466 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 00:59:59.931599 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 00:59:59.931674 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 00:59:59.932004 (+ 330us) server_negotiation.cc:299] Negotiation successful
0926 00:59:59.932073 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 00:59:59.932446 9131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 00:59:59.923354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:45843 (local address 127.3.75.1:59791)
0926 00:59:59.923502 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 00:59:59.923542 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 00:59:59.924656 (+ 1114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 00:59:59.924907 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 00:59:59.924910 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 00:59:59.925152 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0926 00:59:59.925347 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:59.925350 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:59.925982 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:59.925982 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:59.926771 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 00:59:59.926773 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 00:59:59.927200 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 00:59:59.927201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 00:59:59.927297 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 00:59:59.927810 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 00:59:59.927819 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 00:59:59.929283 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 00:59:59.930634 (+ 1351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 00:59:59.930641 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 00:59:59.930644 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 00:59:59.930990 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 00:59:59.931256 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 00:59:59.931257 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 00:59:59.931258 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 00:59:59.931394 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 00:59:59.931678 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 00:59:59.931685 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 00:59:59.931853 (+ 168us) client_negotiation.cc:769] Sending connection context
0926 00:59:59.931975 (+ 122us) client_negotiation.cc:240] Negotiation successful
0926 00:59:59.932151 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":80,"threads_started":1}
I20260926 00:59:59.932968 9129 heartbeater.cc:347] Connected to a master server at 127.3.75.62:45843
I20260926 00:59:59.933077 9129 heartbeater.cc:464] Registering TS with master...
I20260926 00:59:59.933279 9129 heartbeater.cc:511] Master 127.3.75.62:45843 requested a full tablet report, sending...
I20260926 00:59:59.933854 8938 ts_manager.cc:194] Registered new tserver with Master: 29914875df3c4e1eb4dfc87840c2908e (127.3.75.1:36213)
I20260926 00:59:59.934517 8938 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.75.1@KRBTEST.COM'} at 127.3.75.1:59791
WARNING: no policy specified for kudu/127.3.75.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab.
Entry for principal kudu/127.3.75.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab.
I20260926 00:59:59.955606 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.2:0
--local_ip_for_outbound_sockets=127.3.75.2
--webserver_interface=127.3.75.2
--webserver_port=0
--tserver_master_addrs=127.3.75.62:45843
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab
--principal=kudu/127.3.75.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:34621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:00:00.034942 9134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:00:00.035193 9134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:00:00.035231 9134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:00:00.036435 9134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:00:00.036509 9134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:00:00.036554 9134 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.2
I20260926 01:00:00.038628 9134 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:34621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab
--principal=kudu/127.3.75.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.75.2
--webserver_port=0
--tserver_master_addrs=127.3.75.62:45843
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.2
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:00:00.038941 9134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:00:00.039294 9134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:00:00.044607 9140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:00:00.044620 9142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:00:00.044656 9139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:00:00.044975 9134 server_base.cc:1034] running on GCE node
I20260926 01:00:00.045199 9134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:00:00.045452 9134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:00:00.046598 9134 hybrid_clock.cc:648] HybridClock initialized: now 1790384400046568 us; error 48 us; skew 500 ppm
Sep 26 01:00:00 dist-test-slave-7v10 krb5kdc[8893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384400, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:00:00.049248 9134 init.cc:377] Logged in from keytab as kudu/127.3.75.2@KRBTEST.COM (short username kudu)
I20260926 01:00:00.049791 9134 webserver.cc:460] Webserver started at http://127.3.75.2:33919/ using document root <none> and password file <none>
I20260926 01:00:00.050045 9134 fs_manager.cc:359] Metadata directory not provided
I20260926 01:00:00.050091 9134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:00:00.050235 9134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:00:00.051393 9134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/ts-1/data/instance:
uuid: "8af1fe9abc324494a042ac1a31eda0f7"
format_stamp: "Formatted at 2026-09-26 01:00:00 on dist-test-slave-7v10"
server_key: "1536d3ab48e3cdcaa0840fcc41ddd5d7"
server_key_iv: "327b83a5a79c8317bcd35858eaf52221"
server_key_version: "encryptionkey@0"
I20260926 01:00:00.051753 9134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/ts-1/wal/instance:
uuid: "8af1fe9abc324494a042ac1a31eda0f7"
format_stamp: "Formatted at 2026-09-26 01:00:00 on dist-test-slave-7v10"
server_key: "1536d3ab48e3cdcaa0840fcc41ddd5d7"
server_key_iv: "327b83a5a79c8317bcd35858eaf52221"
server_key_version: "encryptionkey@0"
I20260926 01:00:00.053113 9134 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260926 01:00:00.053922 9149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:00.054109 9134 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:00:00.054209 9134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/ts-1/wal
uuid: "8af1fe9abc324494a042ac1a31eda0f7"
format_stamp: "Formatted at 2026-09-26 01:00:00 on dist-test-slave-7v10"
server_key: "1536d3ab48e3cdcaa0840fcc41ddd5d7"
server_key_iv: "327b83a5a79c8317bcd35858eaf52221"
server_key_version: "encryptionkey@0"
I20260926 01:00:00.054306 9134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:00:00.079741 9134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:00:00.082049 9134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:00:00.082222 9134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:00:00.082472 9134 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:00:00.082800 9134 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:00:00.082856 9134 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:00.082917 9134 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:00:00.082960 9134 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:00.088747 9134 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.2:43689
I20260926 01:00:00.088804 9262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.2:43689 every 8 connection(s)
I20260926 01:00:00.089156 9134 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
I20260926 01:00:00.090337 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 9134
I20260926 01:00:00.090471 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/ts-1/wal/instance
I20260926 01:00:00.090639 3372 external_mini_cluster.cc:1428] Setting key 3f1cf98162c9e7e08aae25e66bf7fffd
Sep 26 01:00:00 dist-test-slave-7v10 krb5kdc[8893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384400, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.2@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.75.3@KRBTEST.COM; defaulting to no policy
I20260926 01:00:00.100360 8977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:00.089969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:47323 (local address 127.3.75.62:45843)
0926 01:00:00.090073 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:00.090076 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:00.090267 (+ 191us) server_negotiation.cc:407] Connection header received
0926 01:00:00.091668 (+ 1401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:00.091669 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:00.091682 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:00.091732 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:00.092429 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:00.092981 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:00.094116 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:00.094353 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:00.094436 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:00.097068 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:00.097069 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:00.097070 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:00:00.097078 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:00.098745 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:00.099220 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:00.099221 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:00.099222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:00.099298 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:00.099625 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:00.099626 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:00.099627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:00.099757 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:00.099843 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:00.100142 (+ 299us) server_negotiation.cc:299] Negotiation successful
0926 01:00:00.100260 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 01:00:00.100612 9265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:00.089872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:45843 (local address 127.3.75.2:47323)
0926 01:00:00.090099 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:00.090149 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:00.091548 (+ 1399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:00.091812 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:00.091816 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:00.092112 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:00.092318 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:00.092322 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:00.093084 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:00.093085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:00.093992 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:00.093994 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:00.094455 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:00.094456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:00.094611 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:00.095335 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:00.095344 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:00.096966 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:00:00.098842 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:00.098850 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:00.098852 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:00.099113 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:00.099375 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:00.099376 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:00.099376 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:00.099499 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:00.099833 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:00:00.099840 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:00:00.099995 (+ 155us) client_negotiation.cc:769] Sending connection context
0926 01:00:00.100106 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 01:00:00.100281 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":106,"threads_started":1}
I20260926 01:00:00.101169 9263 heartbeater.cc:347] Connected to a master server at 127.3.75.62:45843
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.3@KRBTEST.COM" created.
I20260926 01:00:00.101279 9263 heartbeater.cc:464] Registering TS with master...
I20260926 01:00:00.101480 9263 heartbeater.cc:511] Master 127.3.75.62:45843 requested a full tablet report, sending...
I20260926 01:00:00.101943 8938 ts_manager.cc:194] Registered new tserver with Master: 8af1fe9abc324494a042ac1a31eda0f7 (127.3.75.2:43689)
I20260926 01:00:00.102581 8938 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.75.2@KRBTEST.COM'} at 127.3.75.2:47323
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab.
Entry for principal kudu/127.3.75.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab.
I20260926 01:00:00.117710 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.3:0
--local_ip_for_outbound_sockets=127.3.75.3
--webserver_interface=127.3.75.3
--webserver_port=0
--tserver_master_addrs=127.3.75.62:45843
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab
--principal=kudu/127.3.75.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:34621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:00:00.198138 9268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:00:00.198375 9268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:00:00.198416 9268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:00:00.199666 9268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:00:00.199736 9268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:00:00.199782 9268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.3
I20260926 01:00:00.201468 9268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:34621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab
--principal=kudu/127.3.75.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.75.3
--webserver_port=0
--tserver_master_addrs=127.3.75.62:45843
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.3
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:00:00.201725 9268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:00:00.201965 9268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:00:00.206784 9274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:00:00.206856 9273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:00:00.206799 9276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:00:00.206943 9268 server_base.cc:1034] running on GCE node
I20260926 01:00:00.207409 9268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:00:00.207655 9268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:00:00.208792 9268 hybrid_clock.cc:648] HybridClock initialized: now 1790384400208781 us; error 29 us; skew 500 ppm
Sep 26 01:00:00 dist-test-slave-7v10 krb5kdc[8893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384400, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:00:00.211186 9268 init.cc:377] Logged in from keytab as kudu/127.3.75.3@KRBTEST.COM (short username kudu)
I20260926 01:00:00.211654 9268 webserver.cc:460] Webserver started at http://127.3.75.3:43931/ using document root <none> and password file <none>
I20260926 01:00:00.211876 9268 fs_manager.cc:359] Metadata directory not provided
I20260926 01:00:00.211926 9268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:00:00.212093 9268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:00:00.213378 9268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/ts-2/data/instance:
uuid: "fb486e774b7c4b758aceff04827e08c4"
format_stamp: "Formatted at 2026-09-26 01:00:00 on dist-test-slave-7v10"
server_key: "092c926ba21eedd98fe387107fb38747"
server_key_iv: "64e43f8b749d06c52d1e398517303df7"
server_key_version: "encryptionkey@0"
I20260926 01:00:00.213845 9268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/ts-2/wal/instance:
uuid: "fb486e774b7c4b758aceff04827e08c4"
format_stamp: "Formatted at 2026-09-26 01:00:00 on dist-test-slave-7v10"
server_key: "092c926ba21eedd98fe387107fb38747"
server_key_iv: "64e43f8b749d06c52d1e398517303df7"
server_key_version: "encryptionkey@0"
I20260926 01:00:00.215157 9268 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260926 01:00:00.215814 9283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:00.216001 9268 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:00:00.216087 9268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/ts-2/wal
uuid: "fb486e774b7c4b758aceff04827e08c4"
format_stamp: "Formatted at 2026-09-26 01:00:00 on dist-test-slave-7v10"
server_key: "092c926ba21eedd98fe387107fb38747"
server_key_iv: "64e43f8b749d06c52d1e398517303df7"
server_key_version: "encryptionkey@0"
I20260926 01:00:00.216176 9268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:00:00.235426 9268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:00:00.237212 9268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:00:00.237385 9268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:00:00.237617 9268 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:00:00.237936 9268 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:00:00.237991 9268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:00.238035 9268 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:00:00.238063 9268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:00.243840 9268 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.3:32967
I20260926 01:00:00.243901 9396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.3:32967 every 8 connection(s)
I20260926 01:00:00.244182 9268 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
Sep 26 01:00:00 dist-test-slave-7v10 krb5kdc[8893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384400, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.3@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 01:00:00.252712 8977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:00.244967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.3:35519 (local address 127.3.75.62:45843)
0926 01:00:00.245025 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:00.245027 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:00.245229 (+ 202us) server_negotiation.cc:407] Connection header received
0926 01:00:00.246114 (+ 885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:00.246115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:00.246124 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:00.246178 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:00.246744 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:00.247227 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:00.248172 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:00.248348 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:00.248396 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:00.250316 (+ 1920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:00.250317 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:00.250318 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:00:00.250324 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:00.251530 (+ 1206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:00.251886 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:00.251887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:00.251887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:00.251936 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:00.252136 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:00.252136 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:00.252136 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:00.252227 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:00.252276 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:00.252498 (+ 222us) server_negotiation.cc:299] Negotiation successful
0926 01:00:00.252608 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 01:00:00.252694 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 9268
I20260926 01:00:00.252729 9399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:00.244922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:45843 (local address 127.3.75.3:35519)
0926 01:00:00.245105 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:00.245144 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:00.246030 (+ 886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:00.246248 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:00.246251 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:00.246493 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:00.246676 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:00.246680 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:00.247296 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:00.247297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:00.248102 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:00.248104 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:00.248414 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:00.248415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:00.248508 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:00.248914 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:00.248922 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:00.250228 (+ 1306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:00:00.251597 (+ 1369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:00.251602 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:00.251604 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:00.251824 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:00.251992 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:00.251993 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:00.251993 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:00.252084 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:00.252280 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:00:00.252284 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:00:00.252394 (+ 110us) client_negotiation.cc:769] Sending connection context
0926 01:00:00.252472 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 01:00:00.252583 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":110,"threads_started":1}
I20260926 01:00:00.252775 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/minicluster-data/ts-2/wal/instance
I20260926 01:00:00.252977 3372 external_mini_cluster.cc:1428] Setting key 2306b8418834c7f3a5c9ad3a5599ad6d
I20260926 01:00:00.253233 9397 heartbeater.cc:347] Connected to a master server at 127.3.75.62:45843
I20260926 01:00:00.253326 9397 heartbeater.cc:464] Registering TS with master...
I20260926 01:00:00.253507 9397 heartbeater.cc:511] Master 127.3.75.62:45843 requested a full tablet report, sending...
I20260926 01:00:00.253871 8938 ts_manager.cc:194] Registered new tserver with Master: fb486e774b7c4b758aceff04827e08c4 (127.3.75.3:32967)
I20260926 01:00:00.254179 8938 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.75.3@KRBTEST.COM'} at 127.3.75.3:35519
I20260926 01:00:00.264554 3372 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/krb5kdc/test-admin.keytab.
Sep 26 01:00:00 dist-test-slave-7v10 krb5kdc[8893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:00:00.283209 3372 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260926 01:00:00.284868 9404 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:00:00.284012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33576 (local address 127.3.75.1:36213)
0926 01:00:00.284167 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:00.284172 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:00.284182 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:00:00.284217 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:00.284218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:00.284232 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:00.284316 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:00.284772 (+ 456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33576: BlockingRecv error: recv got EOF from 127.0.0.1:33576 (error 108)
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":61,"threads_started":1}
W20260926 01:00:00.284919 3372 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/krb5kdc/krb5cc' of type FILE
Sep 26 01:00:00 dist-test-slave-7v10 krb5kdc[8893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:00:00.285943 3372 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 01:00:00 dist-test-slave-7v10 krb5kdc[8893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:00:00.286857 3372 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 01:00:00 dist-test-slave-7v10 krb5kdc[8893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.75.1@KRBTEST.COM
I20260926 01:00:00.294665 9404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:00.287432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33582 (local address 127.3.75.1:36213)
0926 01:00:00.287489 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:00.287491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:00.287554 (+ 63us) server_negotiation.cc:407] Connection header received
0926 01:00:00.287633 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:00.287634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:00.287645 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:00.287701 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:00.288122 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:00.288864 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:00.289617 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:00.289956 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:00.290037 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:00.292062 (+ 2025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:00.292071 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:00.292071 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:00:00.292079 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:00.293442 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:00.293800 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:00.293803 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:00.293803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:00.293865 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:00.294073 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:00.294074 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:00.294074 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:00.294213 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:00.294367 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:00.294461 (+ 94us) server_negotiation.cc:299] Negotiation successful
0926 01:00:00.294542 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260926 01:00:00.296425 9404 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:00:00.295947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33586 (local address 127.3.75.1:36213)
0926 01:00:00.295980 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:00.295982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:00.296022 (+ 40us) server_negotiation.cc:407] Connection header received
0926 01:00:00.296069 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:00.296070 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:00.296078 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:00.296123 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:00.296373 (+ 250us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33586: BlockingRecv error: recv got EOF from 127.0.0.1:33586 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260926 01:00:00.296541 3372 init.cc:244] Error while opening credential cache '/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/krb5kdc/krb5cc)
Sep 26 01:00:00 dist-test-slave-7v10 krb5kdc[8893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:00:00.297505 3372 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 01:00:00 dist-test-slave-7v10 krb5kdc[8893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:00:00.298319 3372 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 01:00:00 dist-test-slave-7v10 krb5kdc[8893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.75.1@KRBTEST.COM
I20260926 01:00:00.305078 9404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:00.298787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33592 (local address 127.3.75.1:36213)
0926 01:00:00.298843 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:00.298845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:00.298915 (+ 70us) server_negotiation.cc:407] Connection header received
0926 01:00:00.298986 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:00.298987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:00.298995 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:00.299051 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:00.299413 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:00.299898 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:00.300650 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:00.300853 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:00.300938 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:00.302786 (+ 1848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:00.302787 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:00.302788 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:00:00.302795 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:00.303995 (+ 1200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:00.304335 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:00.304336 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:00.304336 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:00.304394 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:00.304596 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:00.304596 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:00.304597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:00.304696 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:00.304796 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:00.304913 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 01:00:00.304987 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260926 01:00:00.306622 9404 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:00:00.306059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33608 (local address 127.3.75.1:36213)
0926 01:00:00.306091 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:00.306093 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:00.306124 (+ 31us) server_negotiation.cc:407] Connection header received
0926 01:00:00.306174 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:00.306175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:00.306181 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:00.306227 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:00.306566 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33608: BlockingRecv error: recv got EOF from 127.0.0.1:33608 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260926 01:00:00.306659 3372 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384372073861-3372-0/krb5kdc/krb5cc' of type FILE
Sep 26 01:00:00 dist-test-slave-7v10 krb5kdc[8893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:00:00.307626 3372 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 01:00:00 dist-test-slave-7v10 krb5kdc[8893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:00:00.308444 3372 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 01:00:00 dist-test-slave-7v10 krb5kdc[8893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.75.1@KRBTEST.COM
I20260926 01:00:00.315009 9404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:00.308986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33610 (local address 127.3.75.1:36213)
0926 01:00:00.309038 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:00.309039 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:00.309050 (+ 11us) server_negotiation.cc:407] Connection header received
0926 01:00:00.309136 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:00.309137 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:00.309146 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:00.309202 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:00.309552 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:00.310004 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:00.310676 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:00.310912 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:00.310972 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:00.312780 (+ 1808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:00.312781 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:00.312782 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:00:00.312788 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:00.313891 (+ 1103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:00.314209 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:00.314210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:00.314211 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:00.314268 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:00.314475 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:00.314476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:00.314476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:00.314574 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:00.314673 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:00.314783 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 01:00:00.314854 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 01:00:00.315563 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 9000
I20260926 01:00:00.320302 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 9134
I20260926 01:00:00.325475 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 9268
I20260926 01:00:00.329675 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 8906
2026-09-26T01:00:00Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1890 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:00:00 dist-test-slave-7v10 krb5kdc[9417](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:00:00 dist-test-slave-7v10 krb5kdc[9417](info): set up 2 sockets
Sep 26 01:00:00 dist-test-slave-7v10 krb5kdc[9417](info): commencing operation
krb5kdc: starting...
W20260926 01:00:01.370224 3372 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Sep 26 01:00:01 dist-test-slave-7v10 krb5kdc[9417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:00:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:00:01Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.75.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.75.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.75.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/krb5kdc/oryx_127.3.75.62.keytab.
Entry for principal oryx/127.3.75.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/krb5kdc/oryx_127.3.75.62.keytab.
I20260926 01:00:01.483693 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:42945
--webserver_interface=127.3.75.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/krb5kdc/oryx_127.3.75.62.keytab
--principal=oryx/127.3.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:43605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.75.62:42945
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:00:01.563382 9430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:00:01.563611 9430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:00:01.563651 9430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:00:01.564790 9430 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:00:01.564857 9430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:00:01.564883 9430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:00:01.564915 9430 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:00:01.564944 9430 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:00:01.566372 9430 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:43605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.75.62:42945
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/krb5kdc/oryx_127.3.75.62.keytab
--principal=oryx/127.3.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:42945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:00:01.566646 9430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:00:01.566887 9430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:00:01.571557 9438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:00:01.571592 9435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:00:01.571569 9436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:00:01.571616 9430 server_base.cc:1034] running on GCE node
I20260926 01:00:01.571964 9430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:00:01.572294 9430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:00:01.573458 9430 hybrid_clock.cc:648] HybridClock initialized: now 1790384401573452 us; error 35 us; skew 500 ppm
Sep 26 01:00:01 dist-test-slave-7v10 krb5kdc[9417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384401, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:00:01.575991 9430 init.cc:377] Logged in from keytab as oryx/127.3.75.62@KRBTEST.COM (short username oryx)
I20260926 01:00:01.576426 9430 webserver.cc:460] Webserver started at http://127.3.75.62:38195/ using document root <none> and password file <none>
I20260926 01:00:01.576653 9430 fs_manager.cc:359] Metadata directory not provided
I20260926 01:00:01.576719 9430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:00:01.576869 9430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:00:01.577854 9430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-0/data/instance:
uuid: "d53706d9bbc34ab49aced7cbb4117f20"
format_stamp: "Formatted at 2026-09-26 01:00:01 on dist-test-slave-7v10"
server_key: "dc1664b1a3465a401dd769e07c7ae600"
server_key_iv: "5354d1a8609c9a5a79febf37dcb998f0"
server_key_version: "encryptionkey@0"
I20260926 01:00:01.578202 9430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-0/wal/instance:
uuid: "d53706d9bbc34ab49aced7cbb4117f20"
format_stamp: "Formatted at 2026-09-26 01:00:01 on dist-test-slave-7v10"
server_key: "dc1664b1a3465a401dd769e07c7ae600"
server_key_iv: "5354d1a8609c9a5a79febf37dcb998f0"
server_key_version: "encryptionkey@0"
I20260926 01:00:01.579530 9430 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 01:00:01.580261 9445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:01.580427 9430 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 01:00:01.580525 9430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-0/wal
uuid: "d53706d9bbc34ab49aced7cbb4117f20"
format_stamp: "Formatted at 2026-09-26 01:00:01 on dist-test-slave-7v10"
server_key: "dc1664b1a3465a401dd769e07c7ae600"
server_key_iv: "5354d1a8609c9a5a79febf37dcb998f0"
server_key_version: "encryptionkey@0"
I20260926 01:00:01.580597 9430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:00:01.589289 9430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:00:01.591513 9430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:00:01.591671 9430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:00:01.595521 9430 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.62:42945
I20260926 01:00:01.595554 9497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.62:42945 every 8 connection(s)
I20260926 01:00:01.595887 9430 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
I20260926 01:00:01.596798 9498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:00:01.598093 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 9430
I20260926 01:00:01.598217 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-0/wal/instance
I20260926 01:00:01.598414 3372 external_mini_cluster.cc:1428] Setting key f63c4e9b896c706a37fd43ca5650cc2a
I20260926 01:00:01.599443 9498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d53706d9bbc34ab49aced7cbb4117f20: Bootstrap starting.
I20260926 01:00:01.600023 9498 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d53706d9bbc34ab49aced7cbb4117f20: Neither blocks nor log segments found. Creating new log.
I20260926 01:00:01.600310 9498 log.cc:824] T 00000000000000000000000000000000 P d53706d9bbc34ab49aced7cbb4117f20: Log is configured to *not* fsync() on all Append() calls
I20260926 01:00:01.601079 9498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d53706d9bbc34ab49aced7cbb4117f20: No bootstrap required, opened a new log
I20260926 01:00:01.603420 9498 raft_consensus.cc:348] T 00000000000000000000000000000000 P d53706d9bbc34ab49aced7cbb4117f20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d53706d9bbc34ab49aced7cbb4117f20" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42945 } }
I20260926 01:00:01.603678 9498 raft_consensus.cc:374] T 00000000000000000000000000000000 P d53706d9bbc34ab49aced7cbb4117f20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:00:01.603732 9498 raft_consensus.cc:729] T 00000000000000000000000000000000 P d53706d9bbc34ab49aced7cbb4117f20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d53706d9bbc34ab49aced7cbb4117f20, State: Initialized, Role: FOLLOWER
I20260926 01:00:01.603904 9498 consensus_queue.cc:260] T 00000000000000000000000000000000 P d53706d9bbc34ab49aced7cbb4117f20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d53706d9bbc34ab49aced7cbb4117f20" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42945 } }
I20260926 01:00:01.603996 9498 raft_consensus.cc:388] T 00000000000000000000000000000000 P d53706d9bbc34ab49aced7cbb4117f20 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:00:01.604025 9498 raft_consensus.cc:482] T 00000000000000000000000000000000 P d53706d9bbc34ab49aced7cbb4117f20 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:00:01.604087 9498 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d53706d9bbc34ab49aced7cbb4117f20 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:00:01.604856 9498 raft_consensus.cc:504] T 00000000000000000000000000000000 P d53706d9bbc34ab49aced7cbb4117f20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d53706d9bbc34ab49aced7cbb4117f20" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42945 } }
I20260926 01:00:01.604967 9498 leader_election.cc:302] T 00000000000000000000000000000000 P d53706d9bbc34ab49aced7cbb4117f20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d53706d9bbc34ab49aced7cbb4117f20; no voters:
I20260926 01:00:01.605129 9498 leader_election.cc:288] T 00000000000000000000000000000000 P d53706d9bbc34ab49aced7cbb4117f20 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:00:01.605157 9503 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d53706d9bbc34ab49aced7cbb4117f20 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:00:01.605340 9503 raft_consensus.cc:686] T 00000000000000000000000000000000 P d53706d9bbc34ab49aced7cbb4117f20 [term 1 LEADER]: Becoming Leader. State: Replica: d53706d9bbc34ab49aced7cbb4117f20, State: Running, Role: LEADER
I20260926 01:00:01.605407 9498 sys_catalog.cc:564] T 00000000000000000000000000000000 P d53706d9bbc34ab49aced7cbb4117f20 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:00:01.605487 9503 consensus_queue.cc:237] T 00000000000000000000000000000000 P d53706d9bbc34ab49aced7cbb4117f20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d53706d9bbc34ab49aced7cbb4117f20" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42945 } }
Sep 26 01:00:01 dist-test-slave-7v10 krb5kdc[9417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM
I20260926 01:00:01.605830 9503 sys_catalog.cc:455] T 00000000000000000000000000000000 P d53706d9bbc34ab49aced7cbb4117f20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d53706d9bbc34ab49aced7cbb4117f20. Latest consensus state: current_term: 1 leader_uuid: "d53706d9bbc34ab49aced7cbb4117f20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d53706d9bbc34ab49aced7cbb4117f20" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42945 } } }
I20260926 01:00:01.605894 9503 sys_catalog.cc:458] T 00000000000000000000000000000000 P d53706d9bbc34ab49aced7cbb4117f20 [sys.catalog]: This master's current role is: LEADER
I20260926 01:00:01.605890 9504 sys_catalog.cc:455] T 00000000000000000000000000000000 P d53706d9bbc34ab49aced7cbb4117f20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d53706d9bbc34ab49aced7cbb4117f20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d53706d9bbc34ab49aced7cbb4117f20" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42945 } } }
I20260926 01:00:01.605976 9504 sys_catalog.cc:458] T 00000000000000000000000000000000 P d53706d9bbc34ab49aced7cbb4117f20 [sys.catalog]: This master's current role is: LEADER
I20260926 01:00:01.606284 9509 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:00:01.606822 9509 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:00:01.610062 9501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:01.598990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33680 (local address 127.3.75.62:42945)
0926 01:00:01.599181 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:01.599188 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:01.599211 (+ 23us) server_negotiation.cc:407] Connection header received
0926 01:00:01.599664 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:01.599671 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:01.599722 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:01.599866 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:01.600591 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:01.601956 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:01.602763 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:01.603465 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:01.603563 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:01.606537 (+ 2974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:01.606541 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:01.606542 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:00:01.606553 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:01.608566 (+ 2013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:01.608908 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:01.608911 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:01.608911 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:01.609014 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:01.609196 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:01.609197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:01.609198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:01.609498 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:01.609644 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:01.609736 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 01:00:01.609869 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":92,"threads_started":1}
I20260926 01:00:01.611464 9509 catalog_manager.cc:1347] Generated new cluster ID: fd7906d0e0074f99817ab00b16016fba
I20260926 01:00:01.611534 9509 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:00:01.627100 9509 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:00:01.627643 9509 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:00:01.631428 9509 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d53706d9bbc34ab49aced7cbb4117f20: Generated new TSK 0
I20260926 01:00:01.631600 9509 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.75.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.75.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.75.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/krb5kdc/oryx_127.3.75.1.keytab.
Entry for principal oryx/127.3.75.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/krb5kdc/oryx_127.3.75.1.keytab.
I20260926 01:00:01.661825 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.1:0
--local_ip_for_outbound_sockets=127.3.75.1
--webserver_interface=127.3.75.1
--webserver_port=0
--tserver_master_addrs=127.3.75.62:42945
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/krb5kdc/oryx_127.3.75.1.keytab
--principal=oryx/127.3.75.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:43605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:00:01.741403 9524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:00:01.741621 9524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:00:01.741657 9524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:00:01.742767 9524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:00:01.742835 9524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:00:01.742877 9524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.1
I20260926 01:00:01.744313 9524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:43605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/krb5kdc/oryx_127.3.75.1.keytab
--principal=oryx/127.3.75.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.75.1
--webserver_port=0
--tserver_master_addrs=127.3.75.62:42945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.1
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:00:01.744558 9524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:00:01.744771 9524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:00:01.749517 9530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:00:01.749577 9524 server_base.cc:1034] running on GCE node
W20260926 01:00:01.749564 9532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:00:01.749529 9529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:00:01.749967 9524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:00:01.750247 9524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:00:01.751385 9524 hybrid_clock.cc:648] HybridClock initialized: now 1790384401751376 us; error 27 us; skew 500 ppm
Sep 26 01:00:01 dist-test-slave-7v10 krb5kdc[9417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384401, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.75.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:00:01.753846 9524 init.cc:377] Logged in from keytab as oryx/127.3.75.1@KRBTEST.COM (short username oryx)
I20260926 01:00:01.754254 9524 webserver.cc:460] Webserver started at http://127.3.75.1:39277/ using document root <none> and password file <none>
I20260926 01:00:01.754493 9524 fs_manager.cc:359] Metadata directory not provided
I20260926 01:00:01.754562 9524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:00:01.754709 9524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:00:01.755759 9524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-0/data/instance:
uuid: "c7ac97c190134edfbd953cc9e2501843"
format_stamp: "Formatted at 2026-09-26 01:00:01 on dist-test-slave-7v10"
server_key: "677712872d15669b0eae1611055a6608"
server_key_iv: "94ad6dc5838e087d85fc5cbbdf4bd5a6"
server_key_version: "encryptionkey@0"
I20260926 01:00:01.756132 9524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-0/wal/instance:
uuid: "c7ac97c190134edfbd953cc9e2501843"
format_stamp: "Formatted at 2026-09-26 01:00:01 on dist-test-slave-7v10"
server_key: "677712872d15669b0eae1611055a6608"
server_key_iv: "94ad6dc5838e087d85fc5cbbdf4bd5a6"
server_key_version: "encryptionkey@0"
I20260926 01:00:01.757468 9524 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:00:01.758175 9539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:01.758334 9524 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:00:01.758435 9524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-0/wal
uuid: "c7ac97c190134edfbd953cc9e2501843"
format_stamp: "Formatted at 2026-09-26 01:00:01 on dist-test-slave-7v10"
server_key: "677712872d15669b0eae1611055a6608"
server_key_iv: "94ad6dc5838e087d85fc5cbbdf4bd5a6"
server_key_version: "encryptionkey@0"
I20260926 01:00:01.758519 9524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:00:01.776365 9524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:00:01.778148 9524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:00:01.778301 9524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:00:01.778517 9524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:00:01.778812 9524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:00:01.778867 9524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:01.778970 9524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:00:01.779001 9524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:01.784696 9524 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.1:33259
I20260926 01:00:01.784742 9652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.1:33259 every 8 connection(s)
I20260926 01:00:01.785045 9524 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
I20260926 01:00:01.786794 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 9524
I20260926 01:00:01.786926 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-0/wal/instance
I20260926 01:00:01.787138 3372 external_mini_cluster.cc:1428] Setting key 4d5d38ad073f4cb124843c3b2f704c22
Sep 26 01:00:01 dist-test-slave-7v10 krb5kdc[9417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384401, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.75.1@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM
I20260926 01:00:01.794365 9501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:01.785737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:44231 (local address 127.3.75.62:42945)
0926 01:00:01.785823 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:01.785825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:01.785992 (+ 167us) server_negotiation.cc:407] Connection header received
0926 01:00:01.786479 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:01.786480 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:01.786489 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:01.786539 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:01.787617 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:01.788117 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:01.789103 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:01.789336 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:01.789454 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:01.791587 (+ 2133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:01.791588 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:01.791589 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:00:01.791596 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:01.793012 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:01.793418 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:01.793420 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:01.793420 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:01.793479 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:01.793732 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:01.793733 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:01.793733 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:01.793849 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:01.793924 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:01.794190 (+ 266us) server_negotiation.cc:299] Negotiation successful
0926 01:00:01.794271 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
WARNING: no policy specified for oryx/127.3.75.2@KRBTEST.COM; defaulting to no policy
I20260926 01:00:01.794585 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:01.785680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:42945 (local address 127.3.75.1:44231)
0926 01:00:01.785880 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:01.785915 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:01.786399 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:01.786590 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:01.786593 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:01.787334 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:01.787517 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:01.787522 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:01.788223 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:01.788224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:01.789011 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:01.789013 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:01.789465 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:01.789466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:01.789595 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:01.790111 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:01.790119 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:01.791493 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:00:01.793096 (+ 1603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:01.793103 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:01.793105 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:01.793327 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:01.793555 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:01.793556 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:01.793556 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:01.793659 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:01.793916 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:00:01.793923 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:00:01.794060 (+ 137us) client_negotiation.cc:769] Sending connection context
0926 01:00:01.794170 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 01:00:01.794328 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":102,"threads_started":1}
I20260926 01:00:01.795099 9653 heartbeater.cc:347] Connected to a master server at 127.3.75.62:42945
I20260926 01:00:01.795179 9653 heartbeater.cc:464] Registering TS with master...
I20260926 01:00:01.795364 9653 heartbeater.cc:511] Master 127.3.75.62:42945 requested a full tablet report, sending...
I20260926 01:00:01.795869 9462 ts_manager.cc:194] Registered new tserver with Master: c7ac97c190134edfbd953cc9e2501843 (127.3.75.1:33259)
I20260926 01:00:01.796432 9462 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.75.1@KRBTEST.COM'} at 127.3.75.1:44231
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.75.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.75.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/krb5kdc/oryx_127.3.75.2.keytab.
Entry for principal oryx/127.3.75.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/krb5kdc/oryx_127.3.75.2.keytab.
I20260926 01:00:01.809819 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.2:0
--local_ip_for_outbound_sockets=127.3.75.2
--webserver_interface=127.3.75.2
--webserver_port=0
--tserver_master_addrs=127.3.75.62:42945
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/krb5kdc/oryx_127.3.75.2.keytab
--principal=oryx/127.3.75.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:43605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:00:01.888912 9658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:00:01.889110 9658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:00:01.889132 9658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:00:01.890218 9658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:00:01.890254 9658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:00:01.890280 9658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.2
I20260926 01:00:01.891676 9658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:43605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/krb5kdc/oryx_127.3.75.2.keytab
--principal=oryx/127.3.75.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.75.2
--webserver_port=0
--tserver_master_addrs=127.3.75.62:42945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.2
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:00:01.891861 9658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:00:01.892025 9658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:00:01.896773 9664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:00:01.896894 9658 server_base.cc:1034] running on GCE node
W20260926 01:00:01.896785 9666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:00:01.897006 9663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:00:01.897221 9658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:00:01.897476 9658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:00:01.898639 9658 hybrid_clock.cc:648] HybridClock initialized: now 1790384401898630 us; error 39 us; skew 500 ppm
Sep 26 01:00:01 dist-test-slave-7v10 krb5kdc[9417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384401, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.75.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:00:01.901139 9658 init.cc:377] Logged in from keytab as oryx/127.3.75.2@KRBTEST.COM (short username oryx)
I20260926 01:00:01.901612 9658 webserver.cc:460] Webserver started at http://127.3.75.2:34063/ using document root <none> and password file <none>
I20260926 01:00:01.901847 9658 fs_manager.cc:359] Metadata directory not provided
I20260926 01:00:01.901916 9658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:00:01.902071 9658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:00:01.903148 9658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-1/data/instance:
uuid: "e6356827c4004b85bf0c0009f1e84df5"
format_stamp: "Formatted at 2026-09-26 01:00:01 on dist-test-slave-7v10"
server_key: "8381e702ef832b146e388034a7302154"
server_key_iv: "ac9e6acff4c7b988a8fa49c5f1edf647"
server_key_version: "encryptionkey@0"
I20260926 01:00:01.903523 9658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-1/wal/instance:
uuid: "e6356827c4004b85bf0c0009f1e84df5"
format_stamp: "Formatted at 2026-09-26 01:00:01 on dist-test-slave-7v10"
server_key: "8381e702ef832b146e388034a7302154"
server_key_iv: "ac9e6acff4c7b988a8fa49c5f1edf647"
server_key_version: "encryptionkey@0"
I20260926 01:00:01.904819 9658 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:00:01.905561 9673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:01.905766 9658 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:00:01.905870 9658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-1/wal
uuid: "e6356827c4004b85bf0c0009f1e84df5"
format_stamp: "Formatted at 2026-09-26 01:00:01 on dist-test-slave-7v10"
server_key: "8381e702ef832b146e388034a7302154"
server_key_iv: "ac9e6acff4c7b988a8fa49c5f1edf647"
server_key_version: "encryptionkey@0"
I20260926 01:00:01.905951 9658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:00:01.920078 9658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:00:01.922371 9658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:00:01.922531 9658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:00:01.922747 9658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:00:01.923116 9658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:00:01.923170 9658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:01.923202 9658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:00:01.923233 9658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:01.928946 9658 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.2:44115
I20260926 01:00:01.928987 9786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.2:44115 every 8 connection(s)
I20260926 01:00:01.929275 9658 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
I20260926 01:00:01.934302 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 9658
I20260926 01:00:01.934422 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-1/wal/instance
I20260926 01:00:01.934634 3372 external_mini_cluster.cc:1428] Setting key a9abcd28c5a9013e4412aa1e8d1a0b7e
Sep 26 01:00:01 dist-test-slave-7v10 krb5kdc[9417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384401, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.75.2@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM
I20260926 01:00:01.938638 9789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:01.929929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:42945 (local address 127.3.75.2:39553)
0926 01:00:01.930102 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:01.930141 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:01.930614 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:01.930841 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:01.930845 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:01.931449 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:01.931715 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:01.931722 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:01.932381 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:01.932382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:01.933140 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:01.933142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:01.933471 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:01.933471 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:01.933561 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:01.933967 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:01.933974 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:01.935459 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:00:01.937195 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:01.937201 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:01.937204 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:01.937535 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:01.937726 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:01.937727 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:01.937727 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:01.937872 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:01.938166 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:00:01.938171 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:00:01.938281 (+ 110us) client_negotiation.cc:769] Sending connection context
0926 01:00:01.938371 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 01:00:01.938484 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":103,"threads_started":1}
I20260926 01:00:01.938645 9501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:01.929962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:39553 (local address 127.3.75.62:42945)
0926 01:00:01.930030 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:01.930032 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:01.930214 (+ 182us) server_negotiation.cc:407] Connection header received
0926 01:00:01.930697 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:01.930698 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:01.930706 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:01.930765 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:01.931808 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:01.932300 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:01.933216 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:01.933403 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:01.933451 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:01.935543 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:01.935545 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:01.935546 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:00:01.935553 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:01.937115 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:01.937615 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:01.937615 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:01.937616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:01.937664 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:01.937940 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:01.937941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:01.937941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:01.938102 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:01.938168 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:01.938416 (+ 248us) server_negotiation.cc:299] Negotiation successful
0926 01:00:01.938505 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 01:00:01.939342 9787 heartbeater.cc:347] Connected to a master server at 127.3.75.62:42945
I20260926 01:00:01.939452 9787 heartbeater.cc:464] Registering TS with master...
I20260926 01:00:01.939630 9787 heartbeater.cc:511] Master 127.3.75.62:42945 requested a full tablet report, sending...
I20260926 01:00:01.940038 9462 ts_manager.cc:194] Registered new tserver with Master: e6356827c4004b85bf0c0009f1e84df5 (127.3.75.2:44115)
I20260926 01:00:01.940586 9462 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.75.2@KRBTEST.COM'} at 127.3.75.2:39553
WARNING: no policy specified for oryx/127.3.75.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.75.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.75.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/krb5kdc/oryx_127.3.75.3.keytab.
Entry for principal oryx/127.3.75.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/krb5kdc/oryx_127.3.75.3.keytab.
I20260926 01:00:01.959748 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.3:0
--local_ip_for_outbound_sockets=127.3.75.3
--webserver_interface=127.3.75.3
--webserver_port=0
--tserver_master_addrs=127.3.75.62:42945
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/krb5kdc/oryx_127.3.75.3.keytab
--principal=oryx/127.3.75.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:43605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:00:02.041013 9792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:00:02.041217 9792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:00:02.041242 9792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:00:02.042318 9792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:00:02.042356 9792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:00:02.042380 9792 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.3
I20260926 01:00:02.043838 9792 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:43605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/krb5kdc/oryx_127.3.75.3.keytab
--principal=oryx/127.3.75.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.75.3
--webserver_port=0
--tserver_master_addrs=127.3.75.62:42945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.3
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:00:02.044050 9792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:00:02.044235 9792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:00:02.050118 9800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:00:02.050181 9797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:00:02.050128 9798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:00:02.050194 9792 server_base.cc:1034] running on GCE node
I20260926 01:00:02.050529 9792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:00:02.050761 9792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:00:02.051903 9792 hybrid_clock.cc:648] HybridClock initialized: now 1790384402051890 us; error 38 us; skew 500 ppm
Sep 26 01:00:02 dist-test-slave-7v10 krb5kdc[9417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384402, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.75.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:00:02.057317 9792 init.cc:377] Logged in from keytab as oryx/127.3.75.3@KRBTEST.COM (short username oryx)
I20260926 01:00:02.057847 9792 webserver.cc:460] Webserver started at http://127.3.75.3:33111/ using document root <none> and password file <none>
I20260926 01:00:02.058070 9792 fs_manager.cc:359] Metadata directory not provided
I20260926 01:00:02.058137 9792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:00:02.058283 9792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:00:02.059407 9792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-2/data/instance:
uuid: "17a6cffb4a6a4ab3bc6f18c12a63b38e"
format_stamp: "Formatted at 2026-09-26 01:00:02 on dist-test-slave-7v10"
server_key: "d2b054605d794b9058095306e0192622"
server_key_iv: "bd65a05e532ce454a4231f47512ab9ec"
server_key_version: "encryptionkey@0"
I20260926 01:00:02.059777 9792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-2/wal/instance:
uuid: "17a6cffb4a6a4ab3bc6f18c12a63b38e"
format_stamp: "Formatted at 2026-09-26 01:00:02 on dist-test-slave-7v10"
server_key: "d2b054605d794b9058095306e0192622"
server_key_iv: "bd65a05e532ce454a4231f47512ab9ec"
server_key_version: "encryptionkey@0"
I20260926 01:00:02.061153 9792 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:00:02.061970 9807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:02.062141 9792 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:00:02.062239 9792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-2/wal
uuid: "17a6cffb4a6a4ab3bc6f18c12a63b38e"
format_stamp: "Formatted at 2026-09-26 01:00:02 on dist-test-slave-7v10"
server_key: "d2b054605d794b9058095306e0192622"
server_key_iv: "bd65a05e532ce454a4231f47512ab9ec"
server_key_version: "encryptionkey@0"
I20260926 01:00:02.062325 9792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:00:02.077454 9792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:00:02.080003 9792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:00:02.080176 9792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:00:02.080421 9792 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:00:02.080682 9792 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:00:02.080713 9792 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:02.080735 9792 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:00:02.080749 9792 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:02.086572 9792 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.3:44197
I20260926 01:00:02.086612 9920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.3:44197 every 8 connection(s)
I20260926 01:00:02.087018 9792 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
Sep 26 01:00:02 dist-test-slave-7v10 krb5kdc[9417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384402, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.75.3@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM
I20260926 01:00:02.094525 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 9792
I20260926 01:00:02.094610 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-2/wal/instance
I20260926 01:00:02.094795 3372 external_mini_cluster.cc:1428] Setting key f89a7e4a775361ba7223792cca330c08
I20260926 01:00:02.095817 9501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:02.087734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.3:59975 (local address 127.3.75.62:42945)
0926 01:00:02.087802 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:02.087804 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:02.088003 (+ 199us) server_negotiation.cc:407] Connection header received
0926 01:00:02.088458 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:02.088459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:02.088469 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:02.088517 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:02.089410 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.089888 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.090816 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.091083 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.091134 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:02.093067 (+ 1933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:02.093069 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:02.093069 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:00:02.093075 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:02.094317 (+ 1242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:02.094723 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:02.094724 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:02.094724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:02.094784 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:02.095112 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:02.095113 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:02.095113 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:02.095254 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:02.095328 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:02.095640 (+ 312us) server_negotiation.cc:299] Negotiation successful
0926 01:00:02.095728 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 01:00:02.096004 9923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:02.087713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:42945 (local address 127.3.75.3:59975)
0926 01:00:02.087878 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:02.087913 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:02.088376 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:02.088576 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:02.088579 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:02.089141 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:02.089335 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:02.089340 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.089964 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.089964 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:02.090743 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:02.090745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.091148 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.091149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:02.091265 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:02.091684 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:02.091692 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:02.092985 (+ 1293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:00:02.094385 (+ 1400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:02.094390 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:02.094392 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:02.094628 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:02.094858 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:02.094859 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:02.094859 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:02.095019 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:02.095318 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:00:02.095325 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:00:02.095492 (+ 167us) client_negotiation.cc:769] Sending connection context
0926 01:00:02.095614 (+ 122us) client_negotiation.cc:240] Negotiation successful
0926 01:00:02.095765 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":108,"threads_started":1}
I20260926 01:00:02.096428 9921 heartbeater.cc:347] Connected to a master server at 127.3.75.62:42945
I20260926 01:00:02.096504 9921 heartbeater.cc:464] Registering TS with master...
I20260926 01:00:02.096689 9921 heartbeater.cc:511] Master 127.3.75.62:42945 requested a full tablet report, sending...
I20260926 01:00:02.097048 9462 ts_manager.cc:194] Registered new tserver with Master: 17a6cffb4a6a4ab3bc6f18c12a63b38e (127.3.75.3:44197)
I20260926 01:00:02.097411 9462 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.75.3@KRBTEST.COM'} at 127.3.75.3:59975
I20260926 01:00:02.106954 3372 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.3.75.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.75.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.75.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/krb5kdc/oryx_127.3.75.61.keytab.
Entry for principal oryx/127.3.75.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/krb5kdc/oryx_127.3.75.61.keytab.
I20260926 01:00:02.131493 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.61:39937
--webserver_interface=127.3.75.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/krb5kdc/oryx_127.3.75.61.keytab
--principal=oryx/127.3.75.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:43605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.75.62:42945,127.3.75.61:39937
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:00:02.224265 9926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:00:02.224464 9926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:00:02.224486 9926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:00:02.225543 9926 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:00:02.225580 9926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:00:02.225594 9926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:00:02.225607 9926 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:00:02.225620 9926 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:00:02.227105 9926 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:43605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.75.62:42945,127.3.75.61:39937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/krb5kdc/oryx_127.3.75.61.keytab
--principal=oryx/127.3.75.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.61:39937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.75.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
W20260926 01:00:02.227200 9926 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.3.75.62:42945,127.3.75.61:39937'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260926 01:00:02.227301 9926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:00:02.227458 9926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:00:02.231938 9931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:00:02.232030 9932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:00:02.232152 9926 server_base.cc:1034] running on GCE node
W20260926 01:00:02.232028 9934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:00:02.232437 9926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:00:02.232668 9926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:00:02.233811 9926 hybrid_clock.cc:648] HybridClock initialized: now 1790384402233781 us; error 40 us; skew 500 ppm
Sep 26 01:00:02 dist-test-slave-7v10 krb5kdc[9417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384402, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.75.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:00:02.236274 9926 init.cc:377] Logged in from keytab as oryx/127.3.75.61@KRBTEST.COM (short username oryx)
I20260926 01:00:02.236734 9926 webserver.cc:460] Webserver started at http://127.3.75.61:41781/ using document root <none> and password file <none>
I20260926 01:00:02.236917 9926 fs_manager.cc:359] Metadata directory not provided
I20260926 01:00:02.236958 9926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:00:02.237042 9926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:00:02.238013 9926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-1/data/instance:
uuid: "e19b197b894d43369530f08dabe843db"
format_stamp: "Formatted at 2026-09-26 01:00:02 on dist-test-slave-7v10"
server_key: "f3d13160738d4378aa7df14c67381b98"
server_key_iv: "e75ee4d28d37a2387112f7c68afb14f3"
server_key_version: "encryptionkey@0"
I20260926 01:00:02.238312 9926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-1/wal/instance:
uuid: "e19b197b894d43369530f08dabe843db"
format_stamp: "Formatted at 2026-09-26 01:00:02 on dist-test-slave-7v10"
server_key: "f3d13160738d4378aa7df14c67381b98"
server_key_iv: "e75ee4d28d37a2387112f7c68afb14f3"
server_key_version: "encryptionkey@0"
I20260926 01:00:02.239683 9926 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:00:02.240350 9941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:02.240532 9926 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:00:02.240602 9926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-1/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-1/wal
uuid: "e19b197b894d43369530f08dabe843db"
format_stamp: "Formatted at 2026-09-26 01:00:02 on dist-test-slave-7v10"
server_key: "f3d13160738d4378aa7df14c67381b98"
server_key_iv: "e75ee4d28d37a2387112f7c68afb14f3"
server_key_version: "encryptionkey@0"
I20260926 01:00:02.240690 9926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:00:02.248731 9926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:00:02.250830 9926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:00:02.250988 9926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:00:02.254806 9926 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.61:39937
I20260926 01:00:02.254869 9993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.61:39937 every 8 connection(s)
I20260926 01:00:02.255234 9926 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-1/data/info.pb
I20260926 01:00:02.256268 9994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:00:02.256281 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 9926
I20260926 01:00:02.258635 9994 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42945 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 01:00:02 dist-test-slave-7v10 krb5kdc[9417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.75.61@KRBTEST.COM
Sep 26 01:00:02 dist-test-slave-7v10 krb5kdc[9417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384402, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.75.61@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM
I20260926 01:00:02.268162 9501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:02.259378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33696 (local address 127.3.75.62:42945)
0926 01:00:02.259432 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:02.259435 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:02.259581 (+ 146us) server_negotiation.cc:407] Connection header received
0926 01:00:02.259651 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:02.259652 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:02.259665 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:02.259736 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:02.260277 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.260823 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.262014 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.262372 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.262466 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:02.264817 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:02.264818 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:02.264819 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:00:02.264827 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:02.266578 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:02.267055 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:02.267057 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:02.267057 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:02.267117 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:02.267447 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:02.267448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:02.267448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:02.267588 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:02.267642 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:02.267882 (+ 240us) server_negotiation.cc:299] Negotiation successful
0926 01:00:02.267969 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 01:00:02.268356 9998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:02.259314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:42945 (local address 127.0.0.1:33696)
0926 01:00:02.259490 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:02.259521 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:02.259598 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:02.259757 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:02.259759 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:02.260105 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:02.260185 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:02.260187 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.261037 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.261038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:02.261890 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:02.261893 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.262467 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.262468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:02.262523 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:02.263021 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:02.263029 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:02.264702 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:00:02.266670 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:02.266676 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:02.266678 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:02.266978 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:02.267213 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:02.267214 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:02.267214 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:02.267356 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:02.267667 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:00:02.267672 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:00:02.267743 (+ 71us) client_negotiation.cc:769] Sending connection context
0926 01:00:02.267854 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 01:00:02.268075 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":80,"threads_started":1}
I20260926 01:00:02.268356 9997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:02.256935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41566 (local address 127.3.75.61:39937)
0926 01:00:02.257108 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:02.257116 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:02.257139 (+ 23us) server_negotiation.cc:407] Connection header received
0926 01:00:02.257723 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:02.257729 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:02.257774 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:02.257891 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:02.258690 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.259858 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.260694 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.261323 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.261426 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:02.264387 (+ 2961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:02.264396 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:02.264396 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:00:02.264404 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:02.266436 (+ 2032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:02.266987 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:02.266991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:02.266991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:02.267117 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:02.267377 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:02.267378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:02.267379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:02.267675 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:02.267836 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:02.267934 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 01:00:02.268161 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":124,"server-negotiator.queue_time_us":123,"thread_start_us":85,"threads_started":1}
I20260926 01:00:02.269011 9994 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.75.61" port: 39937 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 01:00:02 dist-test-slave-7v10 krb5kdc[9417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384402, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.75.61@KRBTEST.COM for oryx/127.3.75.61@KRBTEST.COM
I20260926 01:00:02.275213 9998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:02.269263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.61:39937 (local address 127.0.0.1:41570)
0926 01:00:02.269317 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:02.269329 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:02.269378 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:02.269489 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:02.269489 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:02.269614 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:02.269677 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:02.269678 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.270257 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.270258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:02.270798 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:02.270800 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.271194 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.271195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:02.271254 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:02.271603 (+ 349us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:02.271607 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:02.272803 (+ 1196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:00:02.274331 (+ 1528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:02.274332 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:02.274332 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:02.274494 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:02.274659 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:02.274660 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:02.274660 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:02.274707 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:02.274967 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:00:02.274967 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:00:02.275001 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 01:00:02.275060 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 01:00:02.275133 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 01:00:02.275326 9997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:02.269337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41570 (local address 127.3.75.61:39937)
0926 01:00:02.269385 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:02.269387 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:02.269405 (+ 18us) server_negotiation.cc:407] Connection header received
0926 01:00:02.269439 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:02.269440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:02.269447 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:02.269485 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:02.269741 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.270182 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.270909 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.271127 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.271178 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:02.272897 (+ 1719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:02.272898 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:02.272898 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:00:02.272904 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:02.274237 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:02.274558 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:02.274559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:02.274559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:02.274603 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:02.274758 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:02.274759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:02.274759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:02.274848 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:02.274959 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:02.275076 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 01:00:02.275176 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":31}
I20260926 01:00:02.276465 9994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e19b197b894d43369530f08dabe843db: Bootstrap starting.
I20260926 01:00:02.277256 9994 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e19b197b894d43369530f08dabe843db: Neither blocks nor log segments found. Creating new log.
I20260926 01:00:02.277539 9994 log.cc:824] T 00000000000000000000000000000000 P e19b197b894d43369530f08dabe843db: Log is configured to *not* fsync() on all Append() calls
I20260926 01:00:02.278244 9994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e19b197b894d43369530f08dabe843db: No bootstrap required, opened a new log
I20260926 01:00:02.279692 9994 raft_consensus.cc:348] T 00000000000000000000000000000000 P e19b197b894d43369530f08dabe843db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d53706d9bbc34ab49aced7cbb4117f20" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42945 } } peers { permanent_uuid: "e19b197b894d43369530f08dabe843db" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 39937 } }
I20260926 01:00:02.279845 9994 raft_consensus.cc:374] T 00000000000000000000000000000000 P e19b197b894d43369530f08dabe843db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:00:02.279909 9994 raft_consensus.cc:729] T 00000000000000000000000000000000 P e19b197b894d43369530f08dabe843db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e19b197b894d43369530f08dabe843db, State: Initialized, Role: FOLLOWER
I20260926 01:00:02.280058 9994 consensus_queue.cc:260] T 00000000000000000000000000000000 P e19b197b894d43369530f08dabe843db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d53706d9bbc34ab49aced7cbb4117f20" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42945 } } peers { permanent_uuid: "e19b197b894d43369530f08dabe843db" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 39937 } }
I20260926 01:00:02.280356 9994 sys_catalog.cc:564] T 00000000000000000000000000000000 P e19b197b894d43369530f08dabe843db [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:00:02.280331 10000 sys_catalog.cc:455] T 00000000000000000000000000000000 P e19b197b894d43369530f08dabe843db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d53706d9bbc34ab49aced7cbb4117f20" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42945 } } peers { permanent_uuid: "e19b197b894d43369530f08dabe843db" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 39937 } } }
I20260926 01:00:02.280447 10000 sys_catalog.cc:458] T 00000000000000000000000000000000 P e19b197b894d43369530f08dabe843db [sys.catalog]: This master's current role is: FOLLOWER
W20260926 01:00:02.282011 10011 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e19b197b894d43369530f08dabe843db: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:00:02.282066 10011 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:00:02.283819 9926 master_runner.cc:428] Detected that this master e19b197b894d43369530f08dabe843db is joining an existing cluster
I20260926 01:00:02.283879 9926 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.75.61:39937
I20260926 01:00:02.291540 9501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:02.284905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33704 (local address 127.3.75.62:42945)
0926 01:00:02.284951 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:02.284954 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:02.285020 (+ 66us) server_negotiation.cc:407] Connection header received
0926 01:00:02.285095 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:02.285096 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:02.285106 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:02.285155 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:02.285725 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.286242 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.287108 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.287314 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.287399 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:02.288708 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:02.288709 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:02.288710 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:00:02.288717 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:02.290254 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:02.290715 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:02.290717 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:02.290717 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:02.290780 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:02.290984 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:02.290985 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:02.290985 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:02.291100 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:02.291174 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:02.291361 (+ 187us) server_negotiation.cc:299] Negotiation successful
0926 01:00:02.291444 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 01:00:02.291561 10017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:02.284825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:42945 (local address 127.0.0.1:33704)
0926 01:00:02.284964 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:02.284975 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:02.285035 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:02.285250 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:02.285252 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:02.285552 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:02.285624 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:02.285626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.286341 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.286342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:02.287023 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:02.287025 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.287411 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.287412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:02.287476 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:02.288168 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:02.288173 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:02.288614 (+ 441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:00:02.290361 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:02.290362 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:02.290362 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:02.290642 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:02.290848 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:02.290849 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:02.290849 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:02.290934 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:02.291178 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:00:02.291179 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:00:02.291220 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 01:00:02.291317 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 01:00:02.291419 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":79,"threads_started":1}
I20260926 01:00:02.291777 9997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:02.285162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41572 (local address 127.3.75.61:39937)
0926 01:00:02.285199 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:02.285201 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:02.285210 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:00:02.285328 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:02.285329 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:02.285337 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:02.285397 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:02.285857 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.286529 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.287560 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.287775 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.287829 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:02.288852 (+ 1023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:02.288853 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:02.288854 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:00:02.288859 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:02.290670 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:02.291064 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:02.291065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:02.291065 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:02.291111 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:02.291286 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:02.291287 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:02.291288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:02.291417 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:02.291475 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:02.291623 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 01:00:02.291696 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 01:00:02.292007 10018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:02.285018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.61:39937 (local address 127.0.0.1:41572)
0926 01:00:02.285172 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:02.285181 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:02.285246 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:02.285430 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:02.285431 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:02.285699 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:02.285765 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:02.285766 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.286615 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.286616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:02.287473 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:02.287476 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.287850 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.287851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:02.287917 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:02.288330 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:02.288335 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:02.288750 (+ 415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:00:02.290753 (+ 2003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:02.290754 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:02.290754 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:02.290988 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:02.291174 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:02.291174 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:02.291175 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:02.291221 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:02.291497 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:00:02.291497 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:00:02.291527 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 01:00:02.291859 (+ 332us) client_negotiation.cc:240] Negotiation successful
0926 01:00:02.291942 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":66,"threads_started":1}
Sep 26 01:00:02 dist-test-slave-7v10 krb5kdc[9417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384401, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.75.62@KRBTEST.COM for oryx/127.3.75.61@KRBTEST.COM
I20260926 01:00:02.301748 9997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:02.294572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41588 (local address 127.3.75.61:39937)
0926 01:00:02.294611 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:02.294613 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:02.294713 (+ 100us) server_negotiation.cc:407] Connection header received
0926 01:00:02.294820 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:02.294820 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:02.294827 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:02.294933 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:02.295389 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.295808 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.296898 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.297062 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.297106 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:02.299107 (+ 2001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:02.299108 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:02.299108 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:00:02.299115 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:02.300539 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:02.300955 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:02.300956 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:02.300957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:02.301009 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:02.301217 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:02.301218 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:02.301219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:02.301328 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:02.301397 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:02.301586 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 01:00:02.301657 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 01:00:02.301759 10019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:02.294479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.61:39937 (local address 127.0.0.1:41588)
0926 01:00:02.294617 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:02.294660 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:02.294735 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:02.294914 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:02.294917 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:02.295249 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:02.295314 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:02.295316 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.295884 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.295885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:02.296816 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:02.296818 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.297126 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.297127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:02.297208 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:02.297586 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:02.297590 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:02.299024 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:00:02.300640 (+ 1616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:02.300647 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:02.300649 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:02.300849 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:02.301089 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:02.301089 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:02.301090 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:02.301154 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:02.301402 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:00:02.301409 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:00:02.301459 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 01:00:02.301549 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 01:00:02.301662 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":88,"threads_started":1}
I20260926 01:00:02.302314 9462 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.3.75.61:39937: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e19b197b894d43369530f08dabe843db"
member_type: NON_VOTER
last_known_addr {
host: "127.3.75.61"
port: 39937
}
attrs {
promote: true
}
}
dest_uuid: "d53706d9bbc34ab49aced7cbb4117f20"
cas_config_opid_index: -1
I20260926 01:00:02.302646 9462 consensus_queue.cc:237] T 00000000000000000000000000000000 P d53706d9bbc34ab49aced7cbb4117f20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d53706d9bbc34ab49aced7cbb4117f20" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42945 } } peers { permanent_uuid: "e19b197b894d43369530f08dabe843db" member_type: NON_VOTER last_known_addr { host: "127.3.75.61" port: 39937 } attrs { promote: true } }
I20260926 01:00:02.303164 10021 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d53706d9bbc34ab49aced7cbb4117f20 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER e19b197b894d43369530f08dabe843db (127.3.75.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d53706d9bbc34ab49aced7cbb4117f20" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42945 } } peers { permanent_uuid: "e19b197b894d43369530f08dabe843db" member_type: NON_VOTER last_known_addr { host: "127.3.75.61" port: 39937 } attrs { promote: true } } }
I20260926 01:00:02.303464 9968 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e19b197b894d43369530f08dabe843db [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:00:02.304193 10021 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.3.75.61:39937
I20260926 01:00:02.304207 9968 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e19b197b894d43369530f08dabe843db [term 1 FOLLOWER]: Refusing update from remote peer d53706d9bbc34ab49aced7cbb4117f20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260926 01:00:02.304505 9926 master.cc:537] Master@127.3.75.61:39937 shutting down...
I20260926 01:00:02.304486 10020 sys_catalog.cc:455] T 00000000000000000000000000000000 P d53706d9bbc34ab49aced7cbb4117f20 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d53706d9bbc34ab49aced7cbb4117f20" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d53706d9bbc34ab49aced7cbb4117f20" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42945 } } peers { permanent_uuid: "e19b197b894d43369530f08dabe843db" member_type: NON_VOTER last_known_addr { host: "127.3.75.61" port: 39937 } attrs { promote: true } } }
I20260926 01:00:02.304499 10021 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d53706d9bbc34ab49aced7cbb4117f20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e19b197b894d43369530f08dabe843db" member_type: NON_VOTER last_known_addr { host: "127.3.75.61" port: 39937 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:00:02.304580 10020 sys_catalog.cc:458] T 00000000000000000000000000000000 P d53706d9bbc34ab49aced7cbb4117f20 [sys.catalog]: This master's current role is: LEADER
W20260926 01:00:02.305286 9446 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260926 01:00:02.305722 9446 consensus_peers.cc:487] T 00000000000000000000000000000000 P d53706d9bbc34ab49aced7cbb4117f20 -> Peer e19b197b894d43369530f08dabe843db (127.3.75.61:39937): Couldn't send request to peer e19b197b894d43369530f08dabe843db. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:00:02.307241 9926 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e19b197b894d43369530f08dabe843db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:00:02.307332 9926 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e19b197b894d43369530f08dabe843db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:00:02.307384 9926 tablet_replica.cc:330] T 00000000000000000000000000000000 P e19b197b894d43369530f08dabe843db: stopping tablet replica
I20260926 01:00:02.319952 9926 master.cc:552] Master@127.3.75.61:39937 shutdown complete.
I20260926 01:00:02.320015 9926 master_runner.cc:305] Clearing existing system tablet
I20260926 01:00:02.320143 9926 fs_manager.cc:359] Metadata directory not provided
I20260926 01:00:02.320204 9926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:00:02.320802 9926 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.002s
I20260926 01:00:02.321327 10025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:02.321447 9926 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:00:02.321506 9926 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 01:00:02.321530 9926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-1/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-1/wal
uuid: "e19b197b894d43369530f08dabe843db"
format_stamp: "Formatted at 2026-09-26 01:00:02 on dist-test-slave-7v10"
server_key: "f3d13160738d4378aa7df14c67381b98"
server_key_iv: "e75ee4d28d37a2387112f7c68afb14f3"
server_key_version: "encryptionkey@0"
I20260926 01:00:02.322135 9926 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P e19b197b894d43369530f08dabe843db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:00:02.323841 9926 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P e19b197b894d43369530f08dabe843db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 01:00:02.323907 9926 log.cc:1196] T 00000000000000000000000000000000 P e19b197b894d43369530f08dabe843db: Deleting WAL directory at /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260926 01:00:02.324120 9926 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P e19b197b894d43369530f08dabe843db: Deleting consensus metadata
I20260926 01:00:02.324258 9926 master_runner.cc:315] Copying system tablet from 127.3.75.62:42945
I20260926 01:00:02.324910 9926 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P e19b197b894d43369530f08dabe843db: tablet copy: Beginning tablet copy session from remote peer at address 127.3.75.62:42945
I20260926 01:00:02.331175 9501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:02.325280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33706 (local address 127.3.75.62:42945)
0926 01:00:02.325325 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:02.325327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:02.325394 (+ 67us) server_negotiation.cc:407] Connection header received
0926 01:00:02.325475 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:02.325476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:02.325486 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:02.325545 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:02.325887 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.326386 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.327239 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.327454 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.327542 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:02.328508 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:02.328509 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:02.328510 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:00:02.328517 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:02.329998 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:02.330378 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:02.330380 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:02.330380 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:02.330435 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:02.330640 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:02.330641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:02.330641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:02.330759 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:02.330838 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:02.331003 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 01:00:02.331080 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 01:00:02.331192 10031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:02.325197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:42945 (local address 127.0.0.1:33706)
0926 01:00:02.325315 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:02.325342 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:02.325412 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:02.325550 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:02.325551 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:02.325744 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:02.325798 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:02.325799 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.326473 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.326474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:02.327152 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:02.327154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.327544 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.327545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:02.327621 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:02.328110 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:02.328115 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:02.328412 (+ 297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:00:02.330101 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:02.330103 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:02.330103 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:02.330285 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:02.330509 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:02.330510 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:02.330510 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:02.330569 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:02.330832 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:00:02.330834 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:00:02.330904 (+ 70us) client_negotiation.cc:769] Sending connection context
0926 01:00:02.330977 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 01:00:02.331081 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":71,"threads_started":1}
I20260926 01:00:02.331684 9482 tablet_copy_service.cc:140] P d53706d9bbc34ab49aced7cbb4117f20: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer e19b197b894d43369530f08dabe843db ({username='oryx', principal='oryx/127.3.75.61@KRBTEST.COM'} at 127.0.0.1:33706)
I20260926 01:00:02.331763 9482 tablet_copy_service.cc:161] P d53706d9bbc34ab49aced7cbb4117f20: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer e19b197b894d43369530f08dabe843db at {username='oryx', principal='oryx/127.3.75.61@KRBTEST.COM'} at 127.0.0.1:33706: session id = e19b197b894d43369530f08dabe843db-00000000000000000000000000000000
I20260926 01:00:02.332393 9482 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P d53706d9bbc34ab49aced7cbb4117f20: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 01:00:02.333048 9926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 01:00:02.333987 9926 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P e19b197b894d43369530f08dabe843db: tablet copy: Starting download of 0 data blocks...
I20260926 01:00:02.334136 9926 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P e19b197b894d43369530f08dabe843db: tablet copy: Starting download of 1 WAL segments...
I20260926 01:00:02.335033 9926 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P e19b197b894d43369530f08dabe843db: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 01:00:02.335842 9482 tablet_copy_service.cc:342] P d53706d9bbc34ab49aced7cbb4117f20: Request end of tablet copy session e19b197b894d43369530f08dabe843db-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.75.61@KRBTEST.COM'} at 127.0.0.1:33706
I20260926 01:00:02.335902 9482 tablet_copy_service.cc:433] P d53706d9bbc34ab49aced7cbb4117f20: ending tablet copy session e19b197b894d43369530f08dabe843db-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer e19b197b894d43369530f08dabe843db
I20260926 01:00:02.338526 9926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:00:02.339808 10035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:00:02.339866 10032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:00:02.339875 10033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:00:02.343032 9926 server_base.cc:1034] running on GCE node
I20260926 01:00:02.343158 9926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:00:02.343307 9926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:00:02.344537 9926 hybrid_clock.cc:648] HybridClock initialized: now 1790384402344529 us; error 23 us; skew 500 ppm
Sep 26 01:00:02 dist-test-slave-7v10 krb5kdc[9417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384402, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.75.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:00:02.345824 9926 init.cc:377] Logged in from keytab as oryx/127.3.75.61@KRBTEST.COM (short username oryx)
I20260926 01:00:02.346109 9926 webserver.cc:460] Webserver started at http://127.3.75.61:43975/ using document root <none> and password file <none>
I20260926 01:00:02.346246 9926 fs_manager.cc:359] Metadata directory not provided
I20260926 01:00:02.346282 9926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:00:02.346987 9926 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:00:02.347520 10042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:02.347640 9926 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:02.347695 9926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-1/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-1/wal
uuid: "e19b197b894d43369530f08dabe843db"
format_stamp: "Formatted at 2026-09-26 01:00:02 on dist-test-slave-7v10"
server_key: "f3d13160738d4378aa7df14c67381b98"
server_key_iv: "e75ee4d28d37a2387112f7c68afb14f3"
server_key_version: "encryptionkey@0"
I20260926 01:00:02.347754 9926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:00:02.369410 9926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:00:02.371287 9926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:00:02.374635 9926 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.61:39937
I20260926 01:00:02.374655 10094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.61:39937 every 8 connection(s)
I20260926 01:00:02.375032 9926 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-1/data/info.pb
I20260926 01:00:02.375922 10095 sys_catalog.cc:263] Verifying existing consensus state
I20260926 01:00:02.376433 10095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e19b197b894d43369530f08dabe843db: Bootstrap starting.
I20260926 01:00:02.379288 10095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e19b197b894d43369530f08dabe843db: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 01:00:02.379524 10095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e19b197b894d43369530f08dabe843db: Bootstrap complete.
I20260926 01:00:02.379747 10095 raft_consensus.cc:348] T 00000000000000000000000000000000 P e19b197b894d43369530f08dabe843db [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d53706d9bbc34ab49aced7cbb4117f20" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42945 } } peers { permanent_uuid: "e19b197b894d43369530f08dabe843db" member_type: NON_VOTER last_known_addr { host: "127.3.75.61" port: 39937 } attrs { promote: true } }
I20260926 01:00:02.379846 10095 raft_consensus.cc:729] T 00000000000000000000000000000000 P e19b197b894d43369530f08dabe843db [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e19b197b894d43369530f08dabe843db, State: Initialized, Role: LEARNER
I20260926 01:00:02.379884 10095 consensus_queue.cc:260] T 00000000000000000000000000000000 P e19b197b894d43369530f08dabe843db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d53706d9bbc34ab49aced7cbb4117f20" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42945 } } peers { permanent_uuid: "e19b197b894d43369530f08dabe843db" member_type: NON_VOTER last_known_addr { host: "127.3.75.61" port: 39937 } attrs { promote: true } }
I20260926 01:00:02.380050 10095 sys_catalog.cc:564] T 00000000000000000000000000000000 P e19b197b894d43369530f08dabe843db [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:00:02.380079 10098 sys_catalog.cc:455] T 00000000000000000000000000000000 P e19b197b894d43369530f08dabe843db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d53706d9bbc34ab49aced7cbb4117f20" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42945 } } peers { permanent_uuid: "e19b197b894d43369530f08dabe843db" member_type: NON_VOTER last_known_addr { host: "127.3.75.61" port: 39937 } attrs { promote: true } } }
I20260926 01:00:02.380254 10098 sys_catalog.cc:458] T 00000000000000000000000000000000 P e19b197b894d43369530f08dabe843db [sys.catalog]: This master's current role is: LEARNER
I20260926 01:00:02.381570 10109 catalog_manager.cc:1259] Loaded cluster ID: fd7906d0e0074f99817ab00b16016fba
I20260926 01:00:02.381628 10109 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e19b197b894d43369530f08dabe843db: loading cluster ID for follower catalog manager: success
I20260926 01:00:02.382738 10109 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e19b197b894d43369530f08dabe843db: acquiring CA information for follower catalog manager: success
I20260926 01:00:02.383049 10109 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e19b197b894d43369530f08dabe843db: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 01:00:02.426635 10110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:02.420626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41614 (local address 127.3.75.61:39937)
0926 01:00:02.420784 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:02.420786 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:02.420792 (+ 6us) server_negotiation.cc:407] Connection header received
0926 01:00:02.420816 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:02.420816 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:02.420824 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:02.420915 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:02.421476 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.422118 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.422838 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.423102 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.423188 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:02.424359 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:02.424360 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:02.424360 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:00:02.424367 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:02.425478 (+ 1111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:02.425852 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:02.425853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:02.425853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:02.425916 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:02.426130 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:02.426131 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:02.426131 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:02.426235 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:02.426337 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:02.426451 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 01:00:02.426528 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":78,"threads_started":1}
WARNING: no policy specified for oryx/127.3.75.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.75.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.75.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/krb5kdc/oryx_127.3.75.60.keytab.
Entry for principal oryx/127.3.75.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/krb5kdc/oryx_127.3.75.60.keytab.
I20260926 01:00:02.449651 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.60:38525
--webserver_interface=127.3.75.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/krb5kdc/oryx_127.3.75.60.keytab
--principal=oryx/127.3.75.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:43605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.75.62:42945,127.3.75.61:39937,127.3.75.60:38525
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:00:02.530200 10113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:00:02.530421 10113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:00:02.530455 10113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:00:02.531621 10113 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:00:02.531685 10113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:00:02.531711 10113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:00:02.531739 10113 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:00:02.531770 10113 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:00:02.533331 10113 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:43605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.75.62:42945,127.3.75.61:39937,127.3.75.60:38525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/krb5kdc/oryx_127.3.75.60.keytab
--principal=oryx/127.3.75.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.60:38525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.75.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:00:02.533594 10113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:00:02.533810 10113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:00:02.538399 10121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:00:02.538491 10118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:00:02.538625 10113 server_base.cc:1034] running on GCE node
W20260926 01:00:02.538400 10119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:00:02.538962 10113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:00:02.539259 10113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:00:02.540433 10113 hybrid_clock.cc:648] HybridClock initialized: now 1790384402540407 us; error 36 us; skew 500 ppm
Sep 26 01:00:02 dist-test-slave-7v10 krb5kdc[9417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384402, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.75.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:00:02.542770 10113 init.cc:377] Logged in from keytab as oryx/127.3.75.60@KRBTEST.COM (short username oryx)
I20260926 01:00:02.543221 10113 webserver.cc:460] Webserver started at http://127.3.75.60:39923/ using document root <none> and password file <none>
I20260926 01:00:02.543452 10113 fs_manager.cc:359] Metadata directory not provided
I20260926 01:00:02.543519 10113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:00:02.543665 10113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:00:02.544675 10113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-2/data/instance:
uuid: "a945a1b3df7b4b8684f9b198beca1c89"
format_stamp: "Formatted at 2026-09-26 01:00:02 on dist-test-slave-7v10"
server_key: "e6c98a57fbf18bca5423efb5bb0585f8"
server_key_iv: "a5e03c5c5bbcebf06673d9a1cf5be769"
server_key_version: "encryptionkey@0"
I20260926 01:00:02.545025 10113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-2/wal/instance:
uuid: "a945a1b3df7b4b8684f9b198beca1c89"
format_stamp: "Formatted at 2026-09-26 01:00:02 on dist-test-slave-7v10"
server_key: "e6c98a57fbf18bca5423efb5bb0585f8"
server_key_iv: "a5e03c5c5bbcebf06673d9a1cf5be769"
server_key_version: "encryptionkey@0"
I20260926 01:00:02.546285 10113 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:00:02.547132 10128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:02.547305 10113 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:00:02.547403 10113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-2/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-2/wal
uuid: "a945a1b3df7b4b8684f9b198beca1c89"
format_stamp: "Formatted at 2026-09-26 01:00:02 on dist-test-slave-7v10"
server_key: "e6c98a57fbf18bca5423efb5bb0585f8"
server_key_iv: "a5e03c5c5bbcebf06673d9a1cf5be769"
server_key_version: "encryptionkey@0"
I20260926 01:00:02.547483 10113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:00:02.554668 10113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:00:02.556824 10113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:00:02.556977 10113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:00:02.560693 10113 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.60:38525
I20260926 01:00:02.560731 10180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.60:38525 every 8 connection(s)
I20260926 01:00:02.561080 10113 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-2/data/info.pb
I20260926 01:00:02.562131 10181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:00:02.563660 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 10113
I20260926 01:00:02.563671 10181 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42945 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 01:00:02 dist-test-slave-7v10 krb5kdc[9417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384402, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.75.60@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM
Sep 26 01:00:02 dist-test-slave-7v10 krb5kdc[9417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.75.60@KRBTEST.COM
I20260926 01:00:02.573679 9501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:02.564556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33712 (local address 127.3.75.62:42945)
0926 01:00:02.564623 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:02.564625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:02.564636 (+ 11us) server_negotiation.cc:407] Connection header received
0926 01:00:02.565069 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:02.565071 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:02.565083 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:02.565137 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:02.565902 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.566665 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.567911 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.568205 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.568273 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:02.570575 (+ 2302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:02.570577 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:02.570577 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:00:02.570584 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:02.572373 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:02.572757 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:02.572759 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:02.572759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:02.572820 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:02.573055 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:02.573056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:02.573057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:02.573178 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:02.573257 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:02.573508 (+ 251us) server_negotiation.cc:299] Negotiation successful
0926 01:00:02.573589 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 01:00:02.573992 10184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:02.564347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:42945 (local address 127.0.0.1:33712)
0926 01:00:02.564541 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:02.564576 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:02.564971 (+ 395us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:02.565170 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:02.565173 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:02.565549 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:02.565771 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:02.565776 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.566757 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.566758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:02.567806 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:02.567808 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.568291 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.568292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:02.568378 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:02.568873 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:02.568881 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:02.570485 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:00:02.572451 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:02.572456 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:02.572458 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:02.572684 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:02.572889 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:02.572890 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:02.572890 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:02.572977 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:02.573247 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:00:02.573253 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:00:02.573373 (+ 120us) client_negotiation.cc:769] Sending connection context
0926 01:00:02.573471 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 01:00:02.573609 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":63,"threads_started":1}
I20260926 01:00:02.574107 10183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:02.564237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35368 (local address 127.3.75.60:38525)
0926 01:00:02.564423 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:02.564434 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:02.564467 (+ 33us) server_negotiation.cc:407] Connection header received
0926 01:00:02.564957 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:02.564964 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:02.564983 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:02.565096 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:02.565846 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.567010 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.567912 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.568360 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.568416 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:02.571191 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:02.571198 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:02.571199 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:00:02.571206 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:02.572935 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:02.573278 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:02.573281 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:02.573281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:02.573334 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:02.573539 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:02.573540 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:02.573540 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:02.573675 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:02.573829 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:02.573877 (+ 48us) server_negotiation.cc:299] Negotiation successful
0926 01:00:02.573944 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":83,"threads_started":1}
I20260926 01:00:02.574678 10181 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.75.61" port: 39937 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 01:00:02 dist-test-slave-7v10 krb5kdc[9417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384402, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.75.60@KRBTEST.COM for oryx/127.3.75.61@KRBTEST.COM
I20260926 01:00:02.580229 10184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:02.574933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.61:39937 (local address 127.0.0.1:41630)
0926 01:00:02.574968 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:02.574978 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:02.575019 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:02.575122 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:02.575123 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:02.575223 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:02.575266 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:02.575267 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.575825 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.575825 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:02.576296 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:02.576297 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.576609 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.576610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:02.576655 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:02.577046 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:02.577050 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:02.578088 (+ 1038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:00:02.579443 (+ 1355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:02.579443 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:02.579444 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:02.579595 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:02.579757 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:02.579757 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:02.579757 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:02.579809 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:02.580034 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:00:02.580034 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:00:02.580066 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 01:00:02.580113 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 01:00:02.580175 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260926 01:00:02.580245 10110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:02.574988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41630 (local address 127.3.75.61:39937)
0926 01:00:02.575037 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:02.575038 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:02.575045 (+ 7us) server_negotiation.cc:407] Connection header received
0926 01:00:02.575065 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:02.575066 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:02.575076 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:02.575118 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:02.575316 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.575767 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.576364 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.576551 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.576599 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:02.578171 (+ 1572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:02.578172 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:02.578172 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:00:02.578178 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:02.579375 (+ 1197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:02.579656 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:02.579657 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:02.579657 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:02.579702 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:02.579888 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:02.579889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:02.579889 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:02.579987 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:02.580030 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:02.580128 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 01:00:02.580188 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 01:00:02.580696 10181 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.75.60" port: 38525 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 01:00:02 dist-test-slave-7v10 krb5kdc[9417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384402, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.75.60@KRBTEST.COM for oryx/127.3.75.60@KRBTEST.COM
I20260926 01:00:02.585999 10184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:02.581003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.60:38525 (local address 127.0.0.1:35376)
0926 01:00:02.581088 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:02.581096 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:02.581132 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:02.581212 (+ 80us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:02.581212 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:02.581313 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:02.581336 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:02.581337 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.581875 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.581876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:02.582316 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:02.582317 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.582614 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.582615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:02.582650 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:02.583068 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:02.583072 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:02.584117 (+ 1045us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:00:02.585274 (+ 1157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:02.585275 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:02.585276 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:02.585425 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:02.585587 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:02.585587 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:02.585588 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:02.585632 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:02.585791 (+ 159us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:00:02.585791 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:00:02.585820 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 01:00:02.585864 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 01:00:02.585929 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260926 01:00:02.586050 10183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:02.581059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35376 (local address 127.3.75.60:38525)
0926 01:00:02.581090 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:02.581091 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:02.581123 (+ 32us) server_negotiation.cc:407] Connection header received
0926 01:00:02.581165 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:02.581165 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:02.581172 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:02.581210 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:02.581373 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.581817 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.582383 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.582555 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.582602 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:02.584184 (+ 1582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:02.584184 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:02.584185 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:00:02.584190 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:02.585210 (+ 1020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:02.585488 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:02.585488 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:02.585489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:02.585533 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:02.585668 (+ 135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:02.585668 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:02.585669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:02.585744 (+ 75us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:02.585785 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:02.585879 (+ 94us) server_negotiation.cc:299] Negotiation successful
0926 01:00:02.585969 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":16}
I20260926 01:00:02.587222 10181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89: Bootstrap starting.
I20260926 01:00:02.588027 10181 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89: Neither blocks nor log segments found. Creating new log.
I20260926 01:00:02.588299 10181 log.cc:824] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89: Log is configured to *not* fsync() on all Append() calls
I20260926 01:00:02.588964 10181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89: No bootstrap required, opened a new log
I20260926 01:00:02.590317 10181 raft_consensus.cc:348] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d53706d9bbc34ab49aced7cbb4117f20" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42945 } } peers { permanent_uuid: "e19b197b894d43369530f08dabe843db" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 39937 } } peers { permanent_uuid: "a945a1b3df7b4b8684f9b198beca1c89" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 38525 } }
I20260926 01:00:02.590469 10181 raft_consensus.cc:374] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:00:02.590515 10181 raft_consensus.cc:729] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a945a1b3df7b4b8684f9b198beca1c89, State: Initialized, Role: FOLLOWER
I20260926 01:00:02.590662 10181 consensus_queue.cc:260] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d53706d9bbc34ab49aced7cbb4117f20" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42945 } } peers { permanent_uuid: "e19b197b894d43369530f08dabe843db" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 39937 } } peers { permanent_uuid: "a945a1b3df7b4b8684f9b198beca1c89" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 38525 } }
I20260926 01:00:02.590965 10186 sys_catalog.cc:455] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d53706d9bbc34ab49aced7cbb4117f20" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42945 } } peers { permanent_uuid: "e19b197b894d43369530f08dabe843db" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 39937 } } peers { permanent_uuid: "a945a1b3df7b4b8684f9b198beca1c89" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 38525 } } }
I20260926 01:00:02.591048 10181 sys_catalog.cc:564] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:00:02.591075 10186 sys_catalog.cc:458] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 01:00:02.592577 10197 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:00:02.592631 10197 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:00:02.594774 10113 master_runner.cc:428] Detected that this master a945a1b3df7b4b8684f9b198beca1c89 is joining an existing cluster
I20260926 01:00:02.594828 10113 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.75.60:38525
I20260926 01:00:02.603866 9501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:02.596412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33716 (local address 127.3.75.62:42945)
0926 01:00:02.596448 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:02.596450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:02.596549 (+ 99us) server_negotiation.cc:407] Connection header received
0926 01:00:02.596639 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:02.596640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:02.596650 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:02.596681 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:02.597064 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.597763 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.598785 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.599106 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.599147 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:02.600111 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:02.600112 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:02.600112 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:00:02.600119 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:02.601932 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:02.602393 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:02.602394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:02.602394 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:02.602446 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:02.602980 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:02.602981 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:02.602981 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:02.603397 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:02.603448 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:02.603666 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 01:00:02.603742 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 01:00:02.604341 10203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:02.595806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:42945 (local address 127.0.0.1:33716)
0926 01:00:02.596495 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:02.596504 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:02.596569 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:02.596738 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:02.596738 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:02.596921 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:02.596980 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:02.596981 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.597858 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.597859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:02.598686 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:02.598688 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.599227 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.599228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:02.599290 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:02.599691 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:02.599696 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:02.600017 (+ 321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:00:02.602033 (+ 2016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:02.602034 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:02.602034 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:02.602304 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:02.602795 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:02.602796 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:02.602797 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:02.602893 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:02.603504 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:00:02.603504 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:00:02.603548 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 01:00:02.604102 (+ 554us) client_negotiation.cc:240] Negotiation successful
0926 01:00:02.604219 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"thread_start_us":85,"threads_started":1}
I20260926 01:00:02.605333 10205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:02.596009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.61:39937 (local address 127.0.0.1:41644)
0926 01:00:02.598193 (+ 2184us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:02.598204 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:02.598285 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:02.598457 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:02.598458 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:02.598623 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:02.598672 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:02.598674 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.599612 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.599612 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:02.600449 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:02.600452 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.600908 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.600909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:02.600974 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:02.601786 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:02.601791 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:02.602204 (+ 413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:00:02.604083 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:02.604084 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:02.604084 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:02.604338 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:02.604662 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:02.604663 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:02.604663 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:02.604726 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:02.605010 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:00:02.605011 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:00:02.605063 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 01:00:02.605135 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 01:00:02.605230 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2163,"thread_start_us":58,"threads_started":1}
I20260926 01:00:02.605383 10110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:02.596319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41644 (local address 127.3.75.61:39937)
0926 01:00:02.596360 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:02.596362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:02.598260 (+ 1898us) server_negotiation.cc:407] Connection header received
0926 01:00:02.598356 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:02.598356 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:02.598366 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:02.598396 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:02.598754 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.599514 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.600603 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.600819 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.600875 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:02.602295 (+ 1420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:02.602297 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:02.602297 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:00:02.602304 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:02.603987 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:02.604417 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:02.604419 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:02.604419 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:02.604482 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:02.604805 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:02.604806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:02.604806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:02.604937 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:02.605006 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:02.605176 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 01:00:02.605273 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Sep 26 01:00:02 dist-test-slave-7v10 krb5kdc[9417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384401, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.75.62@KRBTEST.COM for oryx/127.3.75.60@KRBTEST.COM
I20260926 01:00:02.622233 10019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:02.606408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.60:38525 (local address 127.0.0.1:35384)
0926 01:00:02.606458 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:02.606468 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:02.606501 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:02.606745 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:02.606746 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:02.606934 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:02.606969 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:02.606970 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.607651 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.607652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:02.608130 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:02.608132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.608569 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.608570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:02.608655 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:02.609045 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:02.609050 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:02.610412 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:00:02.621124 (+ 10712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:02.621125 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:02.621126 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:02.621343 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:02.621608 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:02.621608 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:02.621609 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:02.621676 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:02.621902 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:00:02.621903 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:00:02.621947 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 01:00:02.622036 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 01:00:02.622140 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 01:00:02.622244 10206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:02.606458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35384 (local address 127.3.75.60:38525)
0926 01:00:02.606616 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:02.606617 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:02.606630 (+ 13us) server_negotiation.cc:407] Connection header received
0926 01:00:02.606656 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:02.606656 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:02.606663 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:02.606727 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:02.607045 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.607568 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.608222 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.608491 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.608570 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:02.610509 (+ 1939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:02.610510 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:02.610510 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:00:02.610516 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:02.621034 (+ 10518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:02.621433 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:02.621434 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:02.621434 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:02.621521 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:02.621727 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:02.621728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:02.621728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:02.621835 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:02.621908 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:02.622058 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 01:00:02.622139 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":85,"threads_started":1}
I20260926 01:00:02.622419 10204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:02.595803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.60:38525 (local address 127.0.0.1:35378)
0926 01:00:02.596084 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:02.596095 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:02.596162 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:02.596335 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:02.596335 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:02.596534 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:02.596619 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:02.596620 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.597472 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.597473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:02.598283 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:02.598284 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.598731 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.598732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:02.598789 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:02.599235 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:02.599240 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:02.619265 (+ 20025us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:00:02.621184 (+ 1919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:02.621185 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:02.621185 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:02.621351 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:02.621563 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:02.621564 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:02.621564 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:02.621619 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:02.622128 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:00:02.622129 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:00:02.622163 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 01:00:02.622221 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 01:00:02.622313 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"spinlock_wait_cycles":8448,"thread_start_us":95,"threads_started":1}
I20260926 01:00:02.622656 10183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:02.595988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35378 (local address 127.3.75.60:38525)
0926 01:00:02.596023 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:02.596025 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:02.596140 (+ 115us) server_negotiation.cc:407] Connection header received
0926 01:00:02.596223 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:02.596224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:02.596231 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:02.596256 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:02.596692 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.597385 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.598360 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.598627 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.598658 (+ 31us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:02.619386 (+ 20728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:02.619387 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:02.619388 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:00:02.619394 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:02.621123 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:02.621421 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:02.621422 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:02.621422 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:02.621493 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:02.621912 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:02.621913 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:02.621913 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:02.622040 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:02.622106 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:02.622512 (+ 406us) server_negotiation.cc:299] Negotiation successful
0926 01:00:02.622580 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 01:00:02.622792 9462 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.3.75.60:38525: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "a945a1b3df7b4b8684f9b198beca1c89"
member_type: NON_VOTER
last_known_addr {
host: "127.3.75.60"
port: 38525
}
attrs {
promote: true
}
}
dest_uuid: "d53706d9bbc34ab49aced7cbb4117f20"
cas_config_opid_index: 5
I20260926 01:00:02.623056 9462 consensus_queue.cc:237] T 00000000000000000000000000000000 P d53706d9bbc34ab49aced7cbb4117f20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d53706d9bbc34ab49aced7cbb4117f20" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42945 } } peers { permanent_uuid: "e19b197b894d43369530f08dabe843db" member_type: NON_VOTER last_known_addr { host: "127.3.75.61" port: 39937 } attrs { promote: true } } peers { permanent_uuid: "a945a1b3df7b4b8684f9b198beca1c89" member_type: NON_VOTER last_known_addr { host: "127.3.75.60" port: 38525 } attrs { promote: true } }
I20260926 01:00:02.623410 10020 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d53706d9bbc34ab49aced7cbb4117f20 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a945a1b3df7b4b8684f9b198beca1c89 (127.3.75.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d53706d9bbc34ab49aced7cbb4117f20" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42945 } } peers { permanent_uuid: "e19b197b894d43369530f08dabe843db" member_type: NON_VOTER last_known_addr { host: "127.3.75.61" port: 39937 } attrs { promote: true } } peers { permanent_uuid: "a945a1b3df7b4b8684f9b198beca1c89" member_type: NON_VOTER last_known_addr { host: "127.3.75.60" port: 38525 } attrs { promote: true } } }
I20260926 01:00:02.623605 10155 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:00:02.624373 10020 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.3.75.60:38525
I20260926 01:00:02.624444 10020 sys_catalog.cc:455] T 00000000000000000000000000000000 P d53706d9bbc34ab49aced7cbb4117f20 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d53706d9bbc34ab49aced7cbb4117f20" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d53706d9bbc34ab49aced7cbb4117f20" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42945 } } peers { permanent_uuid: "e19b197b894d43369530f08dabe843db" member_type: NON_VOTER last_known_addr { host: "127.3.75.61" port: 39937 } attrs { promote: true } } peers { permanent_uuid: "a945a1b3df7b4b8684f9b198beca1c89" member_type: NON_VOTER last_known_addr { host: "127.3.75.60" port: 38525 } attrs { promote: true } } }
I20260926 01:00:02.624505 10020 sys_catalog.cc:458] T 00000000000000000000000000000000 P d53706d9bbc34ab49aced7cbb4117f20 [sys.catalog]: This master's current role is: LEADER
I20260926 01:00:02.624632 10155 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89 [term 1 FOLLOWER]: Refusing update from remote peer d53706d9bbc34ab49aced7cbb4117f20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 01:00:02.624802 10113 master.cc:537] Master@127.3.75.60:38525 shutting down...
I20260926 01:00:02.624890 10021 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d53706d9bbc34ab49aced7cbb4117f20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a945a1b3df7b4b8684f9b198beca1c89" member_type: NON_VOTER last_known_addr { host: "127.3.75.60" port: 38525 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 01:00:02.625505 9448 consensus_peers.cc:487] T 00000000000000000000000000000000 P d53706d9bbc34ab49aced7cbb4117f20 -> Peer a945a1b3df7b4b8684f9b198beca1c89 (127.3.75.60:38525): Couldn't send request to peer a945a1b3df7b4b8684f9b198beca1c89. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:00:02.627882 10113 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:00:02.627966 10113 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:00:02.628010 10113 tablet_replica.cc:330] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89: stopping tablet replica
I20260926 01:00:02.628885 10019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:02.623472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.61:39937 (local address 127.0.0.1:41656)
0926 01:00:02.623622 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:02.623633 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:02.623679 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:02.623918 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:02.623920 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:02.623921 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:00:02.623958 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:02.623959 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.626471 (+ 2512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.626472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:02.627701 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:02.627704 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.628624 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.628625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:02.628690 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:02.628710 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 01:00:02.628759 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 01:00:02.628807 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136}
I20260926 01:00:02.629099 10110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:02.623557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41656 (local address 127.3.75.61:39937)
0926 01:00:02.623594 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:02.623596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:02.623730 (+ 134us) server_negotiation.cc:407] Connection header received
0926 01:00:02.623753 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:02.623754 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:02.623777 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:02.623855 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:00:02.624028 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.624735 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.627799 (+ 3064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.628445 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.628507 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:02.628590 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:02.628988 (+ 398us) server_negotiation.cc:299] Negotiation successful
0926 01:00:02.629039 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 01:00:02.629323 10069 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e19b197b894d43369530f08dabe843db [term 1 LEARNER]: Refusing update from remote peer d53706d9bbc34ab49aced7cbb4117f20: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 01:00:02.629521 10021 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d53706d9bbc34ab49aced7cbb4117f20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e19b197b894d43369530f08dabe843db" member_type: NON_VOTER last_known_addr { host: "127.3.75.61" port: 39937 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260926 01:00:02.630041 10069 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e19b197b894d43369530f08dabe843db [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a945a1b3df7b4b8684f9b198beca1c89 (127.3.75.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d53706d9bbc34ab49aced7cbb4117f20" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42945 } } peers { permanent_uuid: "e19b197b894d43369530f08dabe843db" member_type: NON_VOTER last_known_addr { host: "127.3.75.61" port: 39937 } attrs { promote: true } } peers { permanent_uuid: "a945a1b3df7b4b8684f9b198beca1c89" member_type: NON_VOTER last_known_addr { host: "127.3.75.60" port: 38525 } attrs { promote: true } } }
I20260926 01:00:02.631119 10208 sys_catalog.cc:455] T 00000000000000000000000000000000 P e19b197b894d43369530f08dabe843db [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d53706d9bbc34ab49aced7cbb4117f20" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d53706d9bbc34ab49aced7cbb4117f20" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42945 } } peers { permanent_uuid: "e19b197b894d43369530f08dabe843db" member_type: NON_VOTER last_known_addr { host: "127.3.75.61" port: 39937 } attrs { promote: true } } peers { permanent_uuid: "a945a1b3df7b4b8684f9b198beca1c89" member_type: NON_VOTER last_known_addr { host: "127.3.75.60" port: 38525 } attrs { promote: true } } }
I20260926 01:00:02.631232 10208 sys_catalog.cc:458] T 00000000000000000000000000000000 P e19b197b894d43369530f08dabe843db [sys.catalog]: This master's current role is: LEARNER
I20260926 01:00:02.631364 10098 sys_catalog.cc:455] T 00000000000000000000000000000000 P e19b197b894d43369530f08dabe843db [sys.catalog]: SysCatalogTable state changed. Reason: New leader d53706d9bbc34ab49aced7cbb4117f20. Latest consensus state: current_term: 1 leader_uuid: "d53706d9bbc34ab49aced7cbb4117f20" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d53706d9bbc34ab49aced7cbb4117f20" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42945 } } peers { permanent_uuid: "e19b197b894d43369530f08dabe843db" member_type: NON_VOTER last_known_addr { host: "127.3.75.61" port: 39937 } attrs { promote: true } } peers { permanent_uuid: "a945a1b3df7b4b8684f9b198beca1c89" member_type: NON_VOTER last_known_addr { host: "127.3.75.60" port: 38525 } attrs { promote: true } } }
I20260926 01:00:02.631462 10098 sys_catalog.cc:458] T 00000000000000000000000000000000 P e19b197b894d43369530f08dabe843db [sys.catalog]: This master's current role is: LEARNER
I20260926 01:00:02.631678 10021 raft_consensus.cc:1050] T 00000000000000000000000000000000 P d53706d9bbc34ab49aced7cbb4117f20: attempting to promote NON_VOTER e19b197b894d43369530f08dabe843db to VOTER
I20260926 01:00:02.631868 10021 consensus_queue.cc:237] T 00000000000000000000000000000000 P d53706d9bbc34ab49aced7cbb4117f20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d53706d9bbc34ab49aced7cbb4117f20" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42945 } } peers { permanent_uuid: "e19b197b894d43369530f08dabe843db" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 39937 } attrs { promote: false } } peers { permanent_uuid: "a945a1b3df7b4b8684f9b198beca1c89" member_type: NON_VOTER last_known_addr { host: "127.3.75.60" port: 38525 } attrs { promote: true } }
I20260926 01:00:02.632339 10069 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e19b197b894d43369530f08dabe843db [term 1 LEARNER]: Refusing update from remote peer d53706d9bbc34ab49aced7cbb4117f20: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20260926 01:00:02.632513 9448 consensus_peers.cc:487] T 00000000000000000000000000000000 P d53706d9bbc34ab49aced7cbb4117f20 -> Peer a945a1b3df7b4b8684f9b198beca1c89 (127.3.75.60:38525): Couldn't send request to peer a945a1b3df7b4b8684f9b198beca1c89. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:00:02.632566 10021 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d53706d9bbc34ab49aced7cbb4117f20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e19b197b894d43369530f08dabe843db" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 39937 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260926 01:00:02.633193 10113 connection.cc:441] server connection from 127.0.0.1:35368 torn down before Call kudu.master.MasterService.ListTables from 127.0.0.1:35368 (request call id 45) could send its response
I20260926 01:00:02.633522 10021 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d53706d9bbc34ab49aced7cbb4117f20 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e19b197b894d43369530f08dabe843db (127.3.75.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d53706d9bbc34ab49aced7cbb4117f20" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42945 } } peers { permanent_uuid: "e19b197b894d43369530f08dabe843db" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 39937 } attrs { promote: false } } peers { permanent_uuid: "a945a1b3df7b4b8684f9b198beca1c89" member_type: NON_VOTER last_known_addr { host: "127.3.75.60" port: 38525 } attrs { promote: true } } }
I20260926 01:00:02.634279 10021 sys_catalog.cc:455] T 00000000000000000000000000000000 P d53706d9bbc34ab49aced7cbb4117f20 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d53706d9bbc34ab49aced7cbb4117f20" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d53706d9bbc34ab49aced7cbb4117f20" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42945 } } peers { permanent_uuid: "e19b197b894d43369530f08dabe843db" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 39937 } attrs { promote: false } } peers { permanent_uuid: "a945a1b3df7b4b8684f9b198beca1c89" member_type: NON_VOTER last_known_addr { host: "127.3.75.60" port: 38525 } attrs { promote: true } } }
I20260926 01:00:02.634305 10209 sys_catalog.cc:455] T 00000000000000000000000000000000 P d53706d9bbc34ab49aced7cbb4117f20 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d53706d9bbc34ab49aced7cbb4117f20" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d53706d9bbc34ab49aced7cbb4117f20" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42945 } } peers { permanent_uuid: "e19b197b894d43369530f08dabe843db" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 39937 } attrs { promote: false } } peers { permanent_uuid: "a945a1b3df7b4b8684f9b198beca1c89" member_type: NON_VOTER last_known_addr { host: "127.3.75.60" port: 38525 } attrs { promote: true } } }
I20260926 01:00:02.634372 10021 sys_catalog.cc:458] T 00000000000000000000000000000000 P d53706d9bbc34ab49aced7cbb4117f20 [sys.catalog]: This master's current role is: LEADER
I20260926 01:00:02.634382 10209 sys_catalog.cc:458] T 00000000000000000000000000000000 P d53706d9bbc34ab49aced7cbb4117f20 [sys.catalog]: This master's current role is: LEADER
I20260926 01:00:02.634593 10069 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e19b197b894d43369530f08dabe843db [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e19b197b894d43369530f08dabe843db (127.3.75.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d53706d9bbc34ab49aced7cbb4117f20" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42945 } } peers { permanent_uuid: "e19b197b894d43369530f08dabe843db" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 39937 } attrs { promote: false } } peers { permanent_uuid: "a945a1b3df7b4b8684f9b198beca1c89" member_type: NON_VOTER last_known_addr { host: "127.3.75.60" port: 38525 } attrs { promote: true } } }
I20260926 01:00:02.635358 10098 sys_catalog.cc:455] T 00000000000000000000000000000000 P e19b197b894d43369530f08dabe843db [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d53706d9bbc34ab49aced7cbb4117f20" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d53706d9bbc34ab49aced7cbb4117f20" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42945 } } peers { permanent_uuid: "e19b197b894d43369530f08dabe843db" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 39937 } attrs { promote: false } } peers { permanent_uuid: "a945a1b3df7b4b8684f9b198beca1c89" member_type: NON_VOTER last_known_addr { host: "127.3.75.60" port: 38525 } attrs { promote: true } } }
I20260926 01:00:02.635453 10098 sys_catalog.cc:458] T 00000000000000000000000000000000 P e19b197b894d43369530f08dabe843db [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:00:02.644542 10113 master.cc:552] Master@127.3.75.60:38525 shutdown complete.
I20260926 01:00:02.644587 10113 master_runner.cc:305] Clearing existing system tablet
I20260926 01:00:02.644730 10113 fs_manager.cc:359] Metadata directory not provided
I20260926 01:00:02.644774 10113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:00:02.645411 10113 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:00:02.645919 10211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:02.646126 10113 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:00:02.646188 10113 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 01:00:02.646217 10113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-2/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-2/wal
uuid: "a945a1b3df7b4b8684f9b198beca1c89"
format_stamp: "Formatted at 2026-09-26 01:00:02 on dist-test-slave-7v10"
server_key: "e6c98a57fbf18bca5423efb5bb0585f8"
server_key_iv: "a5e03c5c5bbcebf06673d9a1cf5be769"
server_key_version: "encryptionkey@0"
I20260926 01:00:02.646853 10113 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:00:02.648336 10113 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 01:00:02.648397 10113 log.cc:1196] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89: Deleting WAL directory at /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260926 01:00:02.648598 10113 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89: Deleting consensus metadata
I20260926 01:00:02.648721 10113 master_runner.cc:315] Copying system tablet from 127.3.75.62:42945
I20260926 01:00:02.649405 10113 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89: tablet copy: Beginning tablet copy session from remote peer at address 127.3.75.62:42945
I20260926 01:00:02.655825 9501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:02.649806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33730 (local address 127.3.75.62:42945)
0926 01:00:02.649861 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:02.649864 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:02.649972 (+ 108us) server_negotiation.cc:407] Connection header received
0926 01:00:02.650051 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:02.650052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:02.650063 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:02.650120 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:02.650451 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.651035 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.652075 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.652291 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.652366 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:02.653327 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:02.653328 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:02.653328 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:00:02.653336 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:02.654626 (+ 1290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:02.655055 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:02.655057 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:02.655057 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:02.655123 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:02.655314 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:02.655316 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:02.655316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:02.655431 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:02.655508 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:02.655658 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 01:00:02.655732 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 01:00:02.655838 10217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:02.649732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:42945 (local address 127.0.0.1:33730)
0926 01:00:02.649892 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:02.649903 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:02.649987 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:02.650125 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:02.650126 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:02.650316 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:02.650370 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:02.650372 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.651114 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.651115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:02.651972 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:02.651975 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.652364 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.652365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:02.652434 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:02.652942 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:02.652947 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:02.653230 (+ 283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:00:02.654726 (+ 1496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:02.654727 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:02.654728 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:02.654966 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:02.655189 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:02.655189 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:02.655190 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:02.655246 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:02.655499 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:00:02.655500 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:00:02.655544 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 01:00:02.655631 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 01:00:02.655733 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":72,"threads_started":1}
I20260926 01:00:02.656229 9482 tablet_copy_service.cc:140] P d53706d9bbc34ab49aced7cbb4117f20: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer a945a1b3df7b4b8684f9b198beca1c89 ({username='oryx', principal='oryx/127.3.75.60@KRBTEST.COM'} at 127.0.0.1:33730)
I20260926 01:00:02.656286 9482 tablet_copy_service.cc:161] P d53706d9bbc34ab49aced7cbb4117f20: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer a945a1b3df7b4b8684f9b198beca1c89 at {username='oryx', principal='oryx/127.3.75.60@KRBTEST.COM'} at 127.0.0.1:33730: session id = a945a1b3df7b4b8684f9b198beca1c89-00000000000000000000000000000000
I20260926 01:00:02.656742 9482 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P d53706d9bbc34ab49aced7cbb4117f20: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 01:00:02.657197 10113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 01:00:02.658066 10113 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89: tablet copy: Starting download of 0 data blocks...
I20260926 01:00:02.658207 10113 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89: tablet copy: Starting download of 1 WAL segments...
I20260926 01:00:02.659075 10113 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 01:00:02.659893 9482 tablet_copy_service.cc:342] P d53706d9bbc34ab49aced7cbb4117f20: Request end of tablet copy session a945a1b3df7b4b8684f9b198beca1c89-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.75.60@KRBTEST.COM'} at 127.0.0.1:33730
I20260926 01:00:02.659951 9482 tablet_copy_service.cc:433] P d53706d9bbc34ab49aced7cbb4117f20: ending tablet copy session a945a1b3df7b4b8684f9b198beca1c89-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer a945a1b3df7b4b8684f9b198beca1c89
I20260926 01:00:02.662453 10113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:00:02.663766 10219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:00:02.663980 10113 server_base.cc:1034] running on GCE node
W20260926 01:00:02.663800 10218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:00:02.663807 10221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:00:02.664188 10113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:00:02.664335 10113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:00:02.665473 10113 hybrid_clock.cc:648] HybridClock initialized: now 1790384402665458 us; error 26 us; skew 500 ppm
Sep 26 01:00:02 dist-test-slave-7v10 krb5kdc[9417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384402, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.75.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:00:02.666715 10113 init.cc:377] Logged in from keytab as oryx/127.3.75.60@KRBTEST.COM (short username oryx)
I20260926 01:00:02.667035 10113 webserver.cc:460] Webserver started at http://127.3.75.60:33887/ using document root <none> and password file <none>
I20260926 01:00:02.667152 10113 fs_manager.cc:359] Metadata directory not provided
I20260926 01:00:02.667205 10113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:00:02.667786 10113 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:00:02.668286 10228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:02.668403 10113 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:00:02.668448 10113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-2/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-2/wal
uuid: "a945a1b3df7b4b8684f9b198beca1c89"
format_stamp: "Formatted at 2026-09-26 01:00:02 on dist-test-slave-7v10"
server_key: "e6c98a57fbf18bca5423efb5bb0585f8"
server_key_iv: "a5e03c5c5bbcebf06673d9a1cf5be769"
server_key_version: "encryptionkey@0"
I20260926 01:00:02.668500 10113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:00:02.681138 10113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:00:02.683110 10113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:00:02.686899 10113 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.60:38525
I20260926 01:00:02.687124 10280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.60:38525 every 8 connection(s)
I20260926 01:00:02.687333 10113 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/master-2/data/info.pb
I20260926 01:00:02.688146 10281 sys_catalog.cc:263] Verifying existing consensus state
I20260926 01:00:02.688625 10281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89: Bootstrap starting.
I20260926 01:00:02.691376 10281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 01:00:02.691638 10281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89: Bootstrap complete.
I20260926 01:00:02.691859 10281 raft_consensus.cc:348] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d53706d9bbc34ab49aced7cbb4117f20" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42945 } } peers { permanent_uuid: "e19b197b894d43369530f08dabe843db" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 39937 } attrs { promote: false } } peers { permanent_uuid: "a945a1b3df7b4b8684f9b198beca1c89" member_type: NON_VOTER last_known_addr { host: "127.3.75.60" port: 38525 } attrs { promote: true } }
I20260926 01:00:02.691941 10281 raft_consensus.cc:729] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a945a1b3df7b4b8684f9b198beca1c89, State: Initialized, Role: LEARNER
I20260926 01:00:02.691974 10281 consensus_queue.cc:260] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d53706d9bbc34ab49aced7cbb4117f20" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42945 } } peers { permanent_uuid: "e19b197b894d43369530f08dabe843db" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 39937 } attrs { promote: false } } peers { permanent_uuid: "a945a1b3df7b4b8684f9b198beca1c89" member_type: NON_VOTER last_known_addr { host: "127.3.75.60" port: 38525 } attrs { promote: true } }
I20260926 01:00:02.692221 10281 sys_catalog.cc:564] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:00:02.692217 10284 sys_catalog.cc:455] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d53706d9bbc34ab49aced7cbb4117f20" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42945 } } peers { permanent_uuid: "e19b197b894d43369530f08dabe843db" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 39937 } attrs { promote: false } } peers { permanent_uuid: "a945a1b3df7b4b8684f9b198beca1c89" member_type: NON_VOTER last_known_addr { host: "127.3.75.60" port: 38525 } attrs { promote: true } } }
I20260926 01:00:02.692337 10284 sys_catalog.cc:458] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89 [sys.catalog]: This master's current role is: LEARNER
I20260926 01:00:02.693996 10295 catalog_manager.cc:1259] Loaded cluster ID: fd7906d0e0074f99817ab00b16016fba
I20260926 01:00:02.694195 10295 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89: loading cluster ID for follower catalog manager: success
I20260926 01:00:02.695384 10295 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89: acquiring CA information for follower catalog manager: success
I20260926 01:00:02.695706 10295 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 01:00:02.740463 10296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:02.734348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35410 (local address 127.3.75.60:38525)
0926 01:00:02.734483 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:02.734485 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:02.734492 (+ 7us) server_negotiation.cc:407] Connection header received
0926 01:00:02.734513 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:02.734513 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:02.734521 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:02.734597 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:02.735255 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.735888 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.736606 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.736809 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.736888 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:02.738116 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:02.738117 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:02.738117 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:00:02.738124 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:02.739323 (+ 1199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:02.739674 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:02.739675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:02.739676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:02.739733 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:02.739943 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:02.739944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:02.739945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:02.740053 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:02.740153 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:02.740271 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 01:00:02.740352 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":59,"threads_started":1}
I20260926 01:00:02.741034 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 9524
I20260926 01:00:02.745687 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.1:33259
--local_ip_for_outbound_sockets=127.3.75.1
--tserver_master_addrs=127.3.75.62:42945
--webserver_port=39277
--webserver_interface=127.3.75.1
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/krb5kdc/oryx_127.3.75.1.keytab
--principal=oryx/127.3.75.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:43605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.75.62:42945,127.3.75.61:39937
--tserver_master_addrs=127.3.75.62:42945,127.3.75.61:39937,127.3.75.60:38525 with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:00:02.825327 10297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:00:02.825522 10297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:00:02.825544 10297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:00:02.826689 10297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:00:02.826730 10297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:00:02.826788 10297 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.1
I20260926 01:00:02.828226 10297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:43605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/krb5kdc/oryx_127.3.75.1.keytab
--principal=oryx/127.3.75.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.1:33259
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.75.1
--webserver_port=39277
--tserver_master_addrs=127.3.75.62:42945,127.3.75.61:39937,127.3.75.60:38525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.1
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:00:02.828464 10297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:00:02.828707 10297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:00:02.833586 10303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:00:02.833665 10302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:00:02.833751 10305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:00:02.833904 10297 server_base.cc:1034] running on GCE node
I20260926 01:00:02.834070 10297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:00:02.834246 10297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:00:02.835381 10297 hybrid_clock.cc:648] HybridClock initialized: now 1790384402835363 us; error 40 us; skew 500 ppm
Sep 26 01:00:02 dist-test-slave-7v10 krb5kdc[9417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384402, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.75.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:00:02.837669 10297 init.cc:377] Logged in from keytab as oryx/127.3.75.1@KRBTEST.COM (short username oryx)
I20260926 01:00:02.838138 10297 webserver.cc:460] Webserver started at http://127.3.75.1:39277/ using document root <none> and password file <none>
I20260926 01:00:02.838356 10297 fs_manager.cc:359] Metadata directory not provided
I20260926 01:00:02.838439 10297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:00:02.839811 10297 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:00:02.840386 10312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:02.840552 10297 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:00:02.840651 10297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-0/wal
uuid: "c7ac97c190134edfbd953cc9e2501843"
format_stamp: "Formatted at 2026-09-26 01:00:01 on dist-test-slave-7v10"
server_key: "677712872d15669b0eae1611055a6608"
server_key_iv: "94ad6dc5838e087d85fc5cbbdf4bd5a6"
server_key_version: "encryptionkey@0"
I20260926 01:00:02.841001 10297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:00:02.850584 10297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:00:02.852361 10297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:00:02.852519 10297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:00:02.852733 10297 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:00:02.853027 10297 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:00:02.853080 10297 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:02.853119 10297 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:00:02.853147 10297 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:02.858851 10297 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.1:33259
I20260926 01:00:02.858942 10425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.1:33259 every 8 connection(s)
I20260926 01:00:02.859309 10297 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
I20260926 01:00:02.859866 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 10297
I20260926 01:00:02.860039 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 9658
I20260926 01:00:02.868459 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.2:44115
--local_ip_for_outbound_sockets=127.3.75.2
--tserver_master_addrs=127.3.75.62:42945
--webserver_port=34063
--webserver_interface=127.3.75.2
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/krb5kdc/oryx_127.3.75.2.keytab
--principal=oryx/127.3.75.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:43605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.75.62:42945,127.3.75.61:39937
--tserver_master_addrs=127.3.75.62:42945,127.3.75.61:39937,127.3.75.60:38525 with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 01:00:02 dist-test-slave-7v10 krb5kdc[9417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384402, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.75.1@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM
Sep 26 01:00:02 dist-test-slave-7v10 krb5kdc[9417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384402, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.75.1@KRBTEST.COM for oryx/127.3.75.60@KRBTEST.COM
Sep 26 01:00:02 dist-test-slave-7v10 krb5kdc[9417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384402, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.75.1@KRBTEST.COM for oryx/127.3.75.61@KRBTEST.COM
I20260926 01:00:02.875332 9501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:02.860817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:35789 (local address 127.3.75.62:42945)
0926 01:00:02.863547 (+ 2730us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:02.863550 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:02.863562 (+ 12us) server_negotiation.cc:407] Connection header received
0926 01:00:02.863591 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:02.863593 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:02.863604 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:02.863695 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:02.864066 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.864867 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.866414 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.866715 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.866759 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:02.871428 (+ 4669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:02.871429 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:02.871430 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:00:02.871439 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:02.873431 (+ 1992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:02.874139 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:02.874141 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:02.874141 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:02.874215 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:02.874494 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:02.874495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:02.874496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:02.874669 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:02.874723 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:02.875074 (+ 351us) server_negotiation.cc:299] Negotiation successful
0926 01:00:02.875172 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2699}
I20260926 01:00:02.875370 10296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:02.862708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:40295 (local address 127.3.75.60:38525)
0926 01:00:02.864097 (+ 1389us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:02.864100 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:02.864109 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:00:02.864137 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:02.864139 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:02.864154 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:02.864211 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:02.864797 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.865715 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.867098 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.867454 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.867539 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:02.871717 (+ 4178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:02.871718 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:02.871719 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:00:02.871726 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:02.873639 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:02.874216 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:02.874217 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:02.874217 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:02.874277 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:02.874461 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:02.874462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:02.874462 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:02.874587 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:02.874648 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:02.875202 (+ 554us) server_negotiation.cc:299] Negotiation successful
0926 01:00:02.875284 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1361}
I20260926 01:00:02.875433 10432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:02.862653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.61:39937 (local address 127.3.75.1:39753)
0926 01:00:02.863019 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:02.863028 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:02.863101 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:02.863330 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:02.863334 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:02.863629 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:02.863950 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:02.863955 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.864977 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.864977 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:02.866309 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:02.866312 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.866855 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.866855 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:02.867030 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:02.867835 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:02.867841 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:02.871550 (+ 3709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:00:02.873587 (+ 2037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:02.873588 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:02.873589 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:02.873993 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:02.874245 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:02.874245 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:02.874246 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:02.874380 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:02.874681 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:00:02.874682 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:00:02.874845 (+ 163us) client_negotiation.cc:769] Sending connection context
0926 01:00:02.875003 (+ 158us) client_negotiation.cc:240] Negotiation successful
0926 01:00:02.875203 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":51,"thread_start_us":69,"threads_started":1}
I20260926 01:00:02.875658 10431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:02.860649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.60:38525 (local address 127.3.75.1:40295)
0926 01:00:02.860783 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:02.860892 (+ 109us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:02.861114 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:02.864263 (+ 3149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:02.864264 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:02.864504 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:02.864591 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:02.864593 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.865884 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.865885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:02.867001 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:02.867005 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.867536 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.867537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:02.867619 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:02.868042 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:02.868047 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:02.871340 (+ 3293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:00:02.873728 (+ 2388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:02.873729 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:02.873729 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:02.874125 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:02.874343 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:02.874344 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:02.874344 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:02.874393 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:02.874675 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:00:02.874682 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:00:02.874845 (+ 163us) client_negotiation.cc:769] Sending connection context
0926 01:00:02.875115 (+ 270us) client_negotiation.cc:240] Negotiation successful
0926 01:00:02.875541 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":329,"thread_start_us":77,"threads_started":1}
I20260926 01:00:02.875859 10110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:02.862728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:39753 (local address 127.3.75.61:39937)
0926 01:00:02.862789 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:02.862792 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:02.863090 (+ 298us) server_negotiation.cc:407] Connection header received
0926 01:00:02.863168 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:02.863169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:02.863179 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:02.863238 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:02.864065 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.864839 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.866413 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.866725 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.866761 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:02.871636 (+ 4875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:02.871638 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:02.871638 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:00:02.871646 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:02.873488 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:02.874077 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:02.874078 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:02.874078 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:02.874139 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:02.874455 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:02.874456 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:02.874456 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:02.874599 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:02.874657 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:02.875676 (+ 1019us) server_negotiation.cc:299] Negotiation successful
0926 01:00:02.875763 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 01:00:02.876181 10427 heartbeater.cc:347] Connected to a master server at 127.3.75.60:38525
I20260926 01:00:02.876178 10428 heartbeater.cc:347] Connected to a master server at 127.3.75.61:39937
I20260926 01:00:02.876293 10428 heartbeater.cc:464] Registering TS with master...
I20260926 01:00:02.876293 10427 heartbeater.cc:464] Registering TS with master...
I20260926 01:00:02.876508 10427 heartbeater.cc:511] Master 127.3.75.60:38525 requested a full tablet report, sending...
I20260926 01:00:02.876508 10428 heartbeater.cc:511] Master 127.3.75.61:39937 requested a full tablet report, sending...
I20260926 01:00:02.876997 10430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:02.860047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:42945 (local address 127.3.75.1:35789)
0926 01:00:02.860291 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:02.860332 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:02.861099 (+ 767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:02.863713 (+ 2614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:02.863714 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:02.863875 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:02.863949 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:02.863955 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.864980 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.864980 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:02.866309 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:02.866312 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.866852 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.866853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:02.867030 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:02.867780 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:02.867790 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:02.871340 (+ 3550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:00:02.873532 (+ 2192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:02.873539 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:02.873542 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:02.874035 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:02.874307 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:02.874308 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:02.874308 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:02.874405 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:02.874796 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:00:02.874797 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:00:02.874851 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 01:00:02.876698 (+ 1847us) client_negotiation.cc:240] Negotiation successful
0926 01:00:02.876834 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":71,"thread_start_us":82,"threads_started":1}
I20260926 01:00:02.877015 10245 ts_manager.cc:194] Registered new tserver with Master: c7ac97c190134edfbd953cc9e2501843 (127.3.75.1:33259)
I20260926 01:00:02.877015 10059 ts_manager.cc:194] Registered new tserver with Master: c7ac97c190134edfbd953cc9e2501843 (127.3.75.1:33259)
I20260926 01:00:02.877683 10426 heartbeater.cc:347] Connected to a master server at 127.3.75.62:42945
I20260926 01:00:02.877733 10426 heartbeater.cc:464] Registering TS with master...
I20260926 01:00:02.877805 10426 heartbeater.cc:511] Master 127.3.75.62:42945 requested a full tablet report, sending...
I20260926 01:00:02.878153 9462 ts_manager.cc:194] Re-registered known tserver with Master: c7ac97c190134edfbd953cc9e2501843 (127.3.75.1:33259)
I20260926 01:00:02.878779 9462 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.75.1@KRBTEST.COM'} at 127.3.75.1:35789
W20260926 01:00:02.957054 10433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:00:02.957257 10433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:00:02.957281 10433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:00:02.958329 10433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:00:02.958361 10433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:00:02.958386 10433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.2
I20260926 01:00:02.959791 10433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:43605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/krb5kdc/oryx_127.3.75.2.keytab
--principal=oryx/127.3.75.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.2:44115
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.75.2
--webserver_port=34063
--tserver_master_addrs=127.3.75.62:42945,127.3.75.61:39937,127.3.75.60:38525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.2
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:00:02.959971 10433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:00:02.960146 10433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:00:02.965163 10439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:00:02.965222 10438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:00:02.965188 10441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:00:02.965354 10433 server_base.cc:1034] running on GCE node
I20260926 01:00:02.965560 10433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:00:02.965750 10433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:00:02.966899 10433 hybrid_clock.cc:648] HybridClock initialized: now 1790384402966850 us; error 36 us; skew 500 ppm
Sep 26 01:00:02 dist-test-slave-7v10 krb5kdc[9417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384402, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.75.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:00:02.969465 10433 init.cc:377] Logged in from keytab as oryx/127.3.75.2@KRBTEST.COM (short username oryx)
I20260926 01:00:02.969938 10433 webserver.cc:460] Webserver started at http://127.3.75.2:34063/ using document root <none> and password file <none>
I20260926 01:00:02.970184 10433 fs_manager.cc:359] Metadata directory not provided
I20260926 01:00:02.970270 10433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:00:02.971668 10433 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:00:02.972213 10448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:02.972378 10433 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 01:00:02.972473 10433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-1/wal
uuid: "e6356827c4004b85bf0c0009f1e84df5"
format_stamp: "Formatted at 2026-09-26 01:00:01 on dist-test-slave-7v10"
server_key: "8381e702ef832b146e388034a7302154"
server_key_iv: "ac9e6acff4c7b988a8fa49c5f1edf647"
server_key_version: "encryptionkey@0"
I20260926 01:00:02.972797 10433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:00:02.984386 10433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:00:02.986205 10433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:00:02.986316 10433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:00:02.986472 10433 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:00:02.986770 10433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:00:02.986825 10433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:02.986866 10433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:00:02.986917 10433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:02.992480 10433 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.2:44115
I20260926 01:00:02.992571 10561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.2:44115 every 8 connection(s)
I20260926 01:00:02.992899 10433 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
I20260926 01:00:02.996670 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 10433
I20260926 01:00:02.996829 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 9792
Sep 26 01:00:02 dist-test-slave-7v10 krb5kdc[9417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384402, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.75.2@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM
Sep 26 01:00:02 dist-test-slave-7v10 krb5kdc[9417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384402, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.75.2@KRBTEST.COM for oryx/127.3.75.60@KRBTEST.COM
Sep 26 01:00:02 dist-test-slave-7v10 krb5kdc[9417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384402, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.75.2@KRBTEST.COM for oryx/127.3.75.61@KRBTEST.COM
I20260926 01:00:03.004227 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.3:44197
--local_ip_for_outbound_sockets=127.3.75.3
--tserver_master_addrs=127.3.75.62:42945
--webserver_port=33111
--webserver_interface=127.3.75.3
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/krb5kdc/oryx_127.3.75.3.keytab
--principal=oryx/127.3.75.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:43605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.75.62:42945,127.3.75.61:39937
--tserver_master_addrs=127.3.75.62:42945,127.3.75.61:39937,127.3.75.60:38525 with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 01:00:03.005509 9501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:02.993737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:33693 (local address 127.3.75.62:42945)
0926 01:00:02.993789 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:02.993791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:02.994016 (+ 225us) server_negotiation.cc:407] Connection header received
0926 01:00:02.995127 (+ 1111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:02.995129 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:02.995141 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:02.995175 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:02.995884 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.996633 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.997952 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.998259 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.998302 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:03.001915 (+ 3613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:03.001917 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:03.001918 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:00:03.001926 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:03.003853 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:03.004363 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:03.004364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:03.004365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:03.004437 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:03.004719 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:03.004720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:03.004720 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:03.004883 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:03.004927 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:03.005320 (+ 393us) server_negotiation.cc:299] Negotiation successful
0926 01:00:03.005412 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 01:00:03.005517 10296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:02.993676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:56699 (local address 127.3.75.60:38525)
0926 01:00:02.993724 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:02.993726 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:02.993985 (+ 259us) server_negotiation.cc:407] Connection header received
0926 01:00:02.995121 (+ 1136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:02.995123 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:02.995136 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:02.995186 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:02.995900 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.996394 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.997962 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.998321 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.998364 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:03.001915 (+ 3551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:03.001917 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:03.001917 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:00:03.001926 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:03.003787 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:03.004585 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:03.004586 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:03.004587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:03.004639 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:03.004841 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:03.004842 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:03.004843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:03.004998 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:03.005040 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:03.005329 (+ 289us) server_negotiation.cc:299] Negotiation successful
0926 01:00:03.005411 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 01:00:03.006619 10566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:02.993678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.60:38525 (local address 127.3.75.2:56699)
0926 01:00:02.993855 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:02.993898 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:02.995012 (+ 1114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:02.995254 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:02.995259 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:02.995553 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:02.995753 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:02.995757 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.996507 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.996508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:02.997853 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:02.997856 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.998445 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.998446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:02.998552 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:03.000003 (+ 1451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:03.000008 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:03.001797 (+ 1789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:00:03.003871 (+ 2074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:03.003877 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:03.003879 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:03.004266 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:03.004714 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:03.004714 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:03.004715 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:03.004767 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:03.005105 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:00:03.005106 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:00:03.005155 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 01:00:03.006097 (+ 942us) client_negotiation.cc:240] Negotiation successful
0926 01:00:03.006350 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":2,"thread_start_us":90,"threads_started":1}
I20260926 01:00:03.006865 10110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:02.994158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:49033 (local address 127.3.75.61:39937)
0926 01:00:02.994206 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:02.994208 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:02.994429 (+ 221us) server_negotiation.cc:407] Connection header received
0926 01:00:02.995127 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:02.995128 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:02.995141 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:02.995174 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:02.995865 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.996641 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.997953 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.998260 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.998309 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:03.003526 (+ 5217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:03.003528 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:03.003528 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:00:03.003537 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:03.005388 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:03.005983 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:03.005984 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:03.005985 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:03.006060 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:03.006291 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:03.006292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:03.006293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:03.006418 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:03.006462 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:03.006658 (+ 196us) server_negotiation.cc:299] Negotiation successful
0926 01:00:03.006758 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 01:00:03.007259 10568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:02.994084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.61:39937 (local address 127.3.75.2:49033)
0926 01:00:02.994354 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:02.994361 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:02.995036 (+ 675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:02.995260 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:02.995261 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:02.995627 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:02.995754 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:02.995761 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.996736 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.996737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:02.997848 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:02.997851 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.998395 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.998395 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:02.998548 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:02.999320 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:02.999332 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:03.003416 (+ 4084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:00:03.005644 (+ 2228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:03.005646 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:03.005646 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:03.005896 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:03.006150 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:03.006150 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:03.006151 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:03.006215 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:03.006509 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:00:03.006510 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:00:03.006548 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 01:00:03.006997 (+ 449us) client_negotiation.cc:240] Negotiation successful
0926 01:00:03.007104 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":15,"thread_start_us":76,"threads_started":1}
I20260926 01:00:03.007325 10563 heartbeater.cc:347] Connected to a master server at 127.3.75.60:38525
I20260926 01:00:03.007416 10563 heartbeater.cc:464] Registering TS with master...
I20260926 01:00:03.007624 10563 heartbeater.cc:511] Master 127.3.75.60:38525 requested a full tablet report, sending...
I20260926 01:00:03.007805 10567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:02.993678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:42945 (local address 127.3.75.2:33693)
0926 01:00:02.993946 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:02.993957 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:02.995025 (+ 1068us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:02.995256 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:02.995259 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:02.995628 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:02.995754 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:02.995761 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.996826 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.996827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:02.997867 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:02.997868 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:02.998394 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:02.998395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:02.998548 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:02.999498 (+ 950us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:02.999503 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:03.001813 (+ 2310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:00:03.003930 (+ 2117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:03.003931 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:03.003931 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:03.004266 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:03.004505 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:03.004506 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:03.004506 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:03.004641 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:03.004974 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:00:03.004981 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:00:03.005155 (+ 174us) client_negotiation.cc:769] Sending connection context
0926 01:00:03.007062 (+ 1907us) client_negotiation.cc:240] Negotiation successful
0926 01:00:03.007223 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":121,"thread_start_us":71,"threads_started":1}
I20260926 01:00:03.007828 10564 heartbeater.cc:347] Connected to a master server at 127.3.75.61:39937
I20260926 01:00:03.007997 10564 heartbeater.cc:464] Registering TS with master...
I20260926 01:00:03.008070 10564 heartbeater.cc:511] Master 127.3.75.61:39937 requested a full tablet report, sending...
I20260926 01:00:03.008155 10245 ts_manager.cc:194] Registered new tserver with Master: e6356827c4004b85bf0c0009f1e84df5 (127.3.75.2:44115)
I20260926 01:00:03.008402 10059 ts_manager.cc:194] Registered new tserver with Master: e6356827c4004b85bf0c0009f1e84df5 (127.3.75.2:44115)
I20260926 01:00:03.008579 10562 heartbeater.cc:347] Connected to a master server at 127.3.75.62:42945
I20260926 01:00:03.008641 10562 heartbeater.cc:464] Registering TS with master...
I20260926 01:00:03.008721 10562 heartbeater.cc:511] Master 127.3.75.62:42945 requested a full tablet report, sending...
I20260926 01:00:03.008975 9462 ts_manager.cc:194] Re-registered known tserver with Master: e6356827c4004b85bf0c0009f1e84df5 (127.3.75.2:44115)
I20260926 01:00:03.009543 9462 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.75.2@KRBTEST.COM'} at 127.3.75.2:33693
I20260926 01:00:03.085458 10019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:03.082320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.60:38525 (local address 127.0.0.1:35422)
0926 01:00:03.082385 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:03.082399 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:03.082474 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:03.082693 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:03.082694 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:03.082697 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:00:03.082752 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:03.082754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:03.083738 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:03.083739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:03.084499 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:03.084502 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:03.085276 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:03.085277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:03.085323 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:03.085326 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:00:03.085354 (+ 28us) client_negotiation.cc:240] Negotiation successful
0926 01:00:03.085393 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 01:00:03.085489 10296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:03.082419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35422 (local address 127.3.75.60:38525)
0926 01:00:03.082492 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:03.082495 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:03.082509 (+ 14us) server_negotiation.cc:407] Connection header received
0926 01:00:03.082538 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:03.082539 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:03.082563 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:03.082641 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:00:03.082843 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:03.083657 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:03.084592 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:03.085188 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:03.085272 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:03.085345 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:03.085373 (+ 28us) server_negotiation.cc:299] Negotiation successful
0926 01:00:03.085409 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 01:00:03.085872 10255 raft_consensus.cc:1203] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260926 01:00:03.086037 10284 sys_catalog.cc:455] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d53706d9bbc34ab49aced7cbb4117f20. Latest consensus state: current_term: 1 leader_uuid: "d53706d9bbc34ab49aced7cbb4117f20" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d53706d9bbc34ab49aced7cbb4117f20" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42945 } } peers { permanent_uuid: "e19b197b894d43369530f08dabe843db" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 39937 } attrs { promote: false } } peers { permanent_uuid: "a945a1b3df7b4b8684f9b198beca1c89" member_type: NON_VOTER last_known_addr { host: "127.3.75.60" port: 38525 } attrs { promote: true } } }
I20260926 01:00:03.086150 10284 sys_catalog.cc:458] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89 [sys.catalog]: This master's current role is: LEARNER
W20260926 01:00:03.088724 10569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:00:03.088934 10569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:00:03.088977 10569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:00:03.090183 10569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:00:03.090255 10569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:00:03.090301 10569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.3
I20260926 01:00:03.091892 10569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:43605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/krb5kdc/oryx_127.3.75.3.keytab
--principal=oryx/127.3.75.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.3:44197
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.75.3
--webserver_port=33111
--tserver_master_addrs=127.3.75.62:42945,127.3.75.61:39937,127.3.75.60:38525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.3
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:00:03.092141 10569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:00:03.092430 10569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:00:03.097529 10574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:00:03.097568 10569 server_base.cc:1034] running on GCE node
W20260926 01:00:03.097615 10577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:00:03.097537 10575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:00:03.097957 10569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:00:03.098199 10569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:00:03.099365 10569 hybrid_clock.cc:648] HybridClock initialized: now 1790384403099345 us; error 32 us; skew 500 ppm
Sep 26 01:00:03 dist-test-slave-7v10 krb5kdc[9417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384403, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.75.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:00:03.101861 10569 init.cc:377] Logged in from keytab as oryx/127.3.75.3@KRBTEST.COM (short username oryx)
I20260926 01:00:03.102273 10569 webserver.cc:460] Webserver started at http://127.3.75.3:33111/ using document root <none> and password file <none>
I20260926 01:00:03.102507 10569 fs_manager.cc:359] Metadata directory not provided
I20260926 01:00:03.102589 10569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:00:03.103930 10569 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:00:03.104473 10584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:03.104638 10569 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:00:03.104736 10569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-2/wal
uuid: "17a6cffb4a6a4ab3bc6f18c12a63b38e"
format_stamp: "Formatted at 2026-09-26 01:00:02 on dist-test-slave-7v10"
server_key: "d2b054605d794b9058095306e0192622"
server_key_iv: "bd65a05e532ce454a4231f47512ab9ec"
server_key_version: "encryptionkey@0"
I20260926 01:00:03.105077 10569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:00:03.120442 10569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:00:03.122148 10569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:00:03.122303 10569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:00:03.122514 10569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:00:03.122807 10569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:00:03.122859 10569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:03.122937 10569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:00:03.122969 10569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:03.128327 10569 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.3:44197
I20260926 01:00:03.128365 10697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.3:44197 every 8 connection(s)
I20260926 01:00:03.128717 10569 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
I20260926 01:00:03.129972 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 10569
Sep 26 01:00:03 dist-test-slave-7v10 krb5kdc[9417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384403, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.75.3@KRBTEST.COM for oryx/127.3.75.60@KRBTEST.COM
Sep 26 01:00:03 dist-test-slave-7v10 krb5kdc[9417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384403, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.75.3@KRBTEST.COM for oryx/127.3.75.61@KRBTEST.COM
Sep 26 01:00:03 dist-test-slave-7v10 krb5kdc[9417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384403, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.75.3@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM
I20260926 01:00:03.142143 10709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:03.131125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35436 (local address 127.3.75.60:38525)
0926 01:00:03.132941 (+ 1816us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:03.132943 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:03.132950 (+ 7us) server_negotiation.cc:407] Connection header received
0926 01:00:03.132969 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:03.132970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:03.132981 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:03.133046 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:03.133836 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:03.134601 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:03.135833 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:03.136145 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:03.137771 (+ 1626us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:03.137816 (+ 45us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:03.137817 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:03.137818 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:00:03.137822 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:03.140272 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:03.141027 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:03.141029 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:03.141030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:03.141098 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:03.141340 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:03.141342 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:03.141342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:03.141491 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:03.141876 (+ 385us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:03.141923 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 01:00:03.142003 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1779,"thread_start_us":54,"threads_started":1}
I20260926 01:00:03.142635 9501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:03.129973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.3:58119 (local address 127.3.75.62:42945)
0926 01:00:03.130016 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:03.130017 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:03.130114 (+ 97us) server_negotiation.cc:407] Connection header received
0926 01:00:03.130560 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:03.130562 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:03.130573 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:03.130605 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:03.131832 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:03.132566 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:03.133831 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:03.134129 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:03.134166 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:03.138690 (+ 4524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:03.138691 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:03.138692 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:00:03.138698 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:03.140884 (+ 2186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:03.141404 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:03.141406 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:03.141406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:03.141466 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:03.141760 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:03.141761 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:03.141761 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:03.141908 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:03.141952 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:03.142451 (+ 499us) server_negotiation.cc:299] Negotiation successful
0926 01:00:03.142530 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 01:00:03.143567 10702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:03.129780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:42945 (local address 127.3.75.3:58119)
0926 01:00:03.130064 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:03.130072 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:03.130466 (+ 394us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:03.130808 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:03.130809 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:03.131444 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:03.131709 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:03.131715 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:03.132663 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:03.132663 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:03.133739 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:03.133741 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:03.134382 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:03.134383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:03.134443 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:03.134923 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:03.134928 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:03.138618 (+ 3690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:00:03.140998 (+ 2380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:03.141004 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:03.141007 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:03.141304 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:03.141541 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:03.141542 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:03.141542 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:03.141672 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:03.142089 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:00:03.142095 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:00:03.142279 (+ 184us) client_negotiation.cc:769] Sending connection context
0926 01:00:03.143129 (+ 850us) client_negotiation.cc:240] Negotiation successful
0926 01:00:03.143342 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":188,"thread_start_us":70,"threads_started":1}
I20260926 01:00:03.143826 10110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:03.129596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.3:37703 (local address 127.3.75.61:39937)
0926 01:00:03.129909 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:03.129911 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:03.129973 (+ 62us) server_negotiation.cc:407] Connection header received
0926 01:00:03.130559 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:03.130561 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:03.130573 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:03.130605 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:03.131822 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:03.132600 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:03.133721 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:03.133992 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:03.134029 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:03.139032 (+ 5003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:03.139033 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:03.139033 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:00:03.139040 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:03.141570 (+ 2530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:03.142653 (+ 1083us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:03.142654 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:03.142655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:03.142714 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:03.143225 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:03.143226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:03.143227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:03.143379 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:03.143444 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:03.143637 (+ 193us) server_negotiation.cc:299] Negotiation successful
0926 01:00:03.143719 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284}
I20260926 01:00:03.143970 10698 heartbeater.cc:347] Connected to a master server at 127.3.75.62:42945
I20260926 01:00:03.144052 10698 heartbeater.cc:464] Registering TS with master...
I20260926 01:00:03.144224 10698 heartbeater.cc:511] Master 127.3.75.62:42945 requested a full tablet report, sending...
I20260926 01:00:03.144536 10713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:03.135377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41662 (local address 127.3.75.61:39937)
0926 01:00:03.135862 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:03.135864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:03.136015 (+ 151us) server_negotiation.cc:407] Connection header received
0926 01:00:03.136038 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:03.136039 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:03.136050 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:03.136112 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:03.137065 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:03.137847 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:03.138909 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:03.139206 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:03.139263 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:03.140836 (+ 1573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:03.140838 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:03.140838 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:00:03.140845 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:03.143250 (+ 2405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:03.143619 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:03.143620 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:03.143621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:03.143677 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:03.143962 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:03.143963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:03.143964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:03.144081 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:03.144151 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:03.144356 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 01:00:03.144438 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":54,"threads_started":1}
I20260926 01:00:03.144573 9462 ts_manager.cc:194] Re-registered known tserver with Master: 17a6cffb4a6a4ab3bc6f18c12a63b38e (127.3.75.3:44197)
I20260926 01:00:03.144766 10704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:03.129501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.61:39937 (local address 127.3.75.3:37703)
0926 01:00:03.129895 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:03.129907 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:03.130466 (+ 559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:03.130794 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:03.130798 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:03.131438 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:03.131709 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:03.131715 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:03.132688 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:03.132688 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:03.133631 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:03.133633 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:03.134099 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:03.134100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:03.134160 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:03.135058 (+ 898us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:03.135063 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:03.138558 (+ 3495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:00:03.142364 (+ 3806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:03.142365 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:03.142365 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:03.142560 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:03.143093 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:03.143094 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:03.143094 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:03.143160 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:03.143475 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:00:03.143476 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:00:03.143516 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 01:00:03.144371 (+ 855us) client_negotiation.cc:240] Negotiation successful
0926 01:00:03.144472 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"mutex_wait_us":73,"thread_start_us":84,"threads_started":1}
I20260926 01:00:03.145010 9462 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.75.3@KRBTEST.COM'} at 127.3.75.3:58119
I20260926 01:00:03.145656 10700 heartbeater.cc:347] Connected to a master server at 127.3.75.61:39937
I20260926 01:00:03.145709 10700 heartbeater.cc:464] Registering TS with master...
I20260926 01:00:03.145758 10700 heartbeater.cc:511] Master 127.3.75.61:39937 requested a full tablet report, sending...
I20260926 01:00:03.146031 10059 ts_manager.cc:194] Registered new tserver with Master: 17a6cffb4a6a4ab3bc6f18c12a63b38e (127.3.75.3:44197)
I20260926 01:00:03.146739 10296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:03.129486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.3:40641 (local address 127.3.75.60:38525)
0926 01:00:03.129602 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:03.129605 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:03.129821 (+ 216us) server_negotiation.cc:407] Connection header received
0926 01:00:03.130572 (+ 751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:03.130573 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:03.130587 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:03.130621 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:03.131804 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:03.132296 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:03.133547 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:03.133856 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:03.133894 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:03.143257 (+ 9363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:03.143259 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:03.143259 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:00:03.143266 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:03.144927 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:03.145965 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:03.145966 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:03.145966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:03.146025 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:03.146225 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:03.146225 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:03.146226 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:03.146335 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:03.146416 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:03.146564 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 01:00:03.146640 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 01:00:03.146759 10703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:03.129467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.60:38525 (local address 127.3.75.3:40641)
0926 01:00:03.129674 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:03.129714 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:03.130447 (+ 733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:03.130794 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:03.130799 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:03.131507 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:03.131709 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:03.131713 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:03.132403 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:03.132404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:03.133435 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:03.133438 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:03.133975 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:03.133975 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:03.134115 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:03.134859 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:03.134886 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:03.138438 (+ 3552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:00:03.145017 (+ 6579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:03.145018 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:03.145018 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:03.145233 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:03.146104 (+ 871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:03.146105 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:03.146106 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:03.146168 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:03.146409 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:00:03.146409 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:00:03.146451 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 01:00:03.146541 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 01:00:03.146641 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":18,"thread_start_us":88,"threads_started":1}
I20260926 01:00:03.147166 10699 heartbeater.cc:347] Connected to a master server at 127.3.75.60:38525
I20260926 01:00:03.147222 10699 heartbeater.cc:464] Registering TS with master...
I20260926 01:00:03.147289 10699 heartbeater.cc:511] Master 127.3.75.60:38525 requested a full tablet report, sending...
I20260926 01:00:03.147535 10245 ts_manager.cc:194] Registered new tserver with Master: 17a6cffb4a6a4ab3bc6f18c12a63b38e (127.3.75.3:44197)
I20260926 01:00:03.147975 10711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:03.134711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33742 (local address 127.3.75.62:42945)
0926 01:00:03.135753 (+ 1042us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:03.135755 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:03.135761 (+ 6us) server_negotiation.cc:407] Connection header received
0926 01:00:03.135784 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:03.135784 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:03.135805 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:03.136432 (+ 627us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:03.136708 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:03.137562 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:03.139426 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:03.139758 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:03.140096 (+ 338us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:03.142682 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:03.142683 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:03.142684 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:00:03.142691 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:03.146713 (+ 4022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:03.147125 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:03.147127 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:03.147127 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:03.147202 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:03.147425 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:03.147426 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:03.147426 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:03.147533 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:03.147636 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:03.147767 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 01:00:03.147850 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1025,"thread_start_us":69,"threads_started":1}
I20260926 01:00:03.149312 9462 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33742:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:00:03.150707 9462 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:00:03.151501 10717 sys_catalog.cc:455] T 00000000000000000000000000000000 P d53706d9bbc34ab49aced7cbb4117f20 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d53706d9bbc34ab49aced7cbb4117f20" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d53706d9bbc34ab49aced7cbb4117f20" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42945 } } peers { permanent_uuid: "e19b197b894d43369530f08dabe843db" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 39937 } attrs { promote: false } } peers { permanent_uuid: "a945a1b3df7b4b8684f9b198beca1c89" member_type: NON_VOTER last_known_addr { host: "127.3.75.60" port: 38525 } attrs { promote: true } } }
I20260926 01:00:03.151631 10717 sys_catalog.cc:458] T 00000000000000000000000000000000 P d53706d9bbc34ab49aced7cbb4117f20 [sys.catalog]: This master's current role is: LEADER
I20260926 01:00:03.152751 10716 raft_consensus.cc:1050] T 00000000000000000000000000000000 P d53706d9bbc34ab49aced7cbb4117f20: attempting to promote NON_VOTER a945a1b3df7b4b8684f9b198beca1c89 to VOTER
I20260926 01:00:03.152896 10716 consensus_queue.cc:237] T 00000000000000000000000000000000 P d53706d9bbc34ab49aced7cbb4117f20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d53706d9bbc34ab49aced7cbb4117f20" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42945 } } peers { permanent_uuid: "e19b197b894d43369530f08dabe843db" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 39937 } attrs { promote: false } } peers { permanent_uuid: "a945a1b3df7b4b8684f9b198beca1c89" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 38525 } attrs { promote: false } }
I20260926 01:00:03.153246 10255 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89 [term 1 LEARNER]: Refusing update from remote peer d53706d9bbc34ab49aced7cbb4117f20: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260926 01:00:03.153452 10717 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d53706d9bbc34ab49aced7cbb4117f20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a945a1b3df7b4b8684f9b198beca1c89" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 38525 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260926 01:00:03.153704 10069 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e19b197b894d43369530f08dabe843db [term 1 FOLLOWER]: Refusing update from remote peer d53706d9bbc34ab49aced7cbb4117f20: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260926 01:00:03.153929 10717 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d53706d9bbc34ab49aced7cbb4117f20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e19b197b894d43369530f08dabe843db" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 39937 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260926 01:00:03.154124 10717 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d53706d9bbc34ab49aced7cbb4117f20 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, a945a1b3df7b4b8684f9b198beca1c89 (127.3.75.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d53706d9bbc34ab49aced7cbb4117f20" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42945 } } peers { permanent_uuid: "e19b197b894d43369530f08dabe843db" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 39937 } attrs { promote: false } } peers { permanent_uuid: "a945a1b3df7b4b8684f9b198beca1c89" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 38525 } attrs { promote: false } } }
I20260926 01:00:03.154261 10255 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, a945a1b3df7b4b8684f9b198beca1c89 (127.3.75.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d53706d9bbc34ab49aced7cbb4117f20" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42945 } } peers { permanent_uuid: "e19b197b894d43369530f08dabe843db" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 39937 } attrs { promote: false } } peers { permanent_uuid: "a945a1b3df7b4b8684f9b198beca1c89" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 38525 } attrs { promote: false } } }
I20260926 01:00:03.154649 10069 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e19b197b894d43369530f08dabe843db [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, a945a1b3df7b4b8684f9b198beca1c89 (127.3.75.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d53706d9bbc34ab49aced7cbb4117f20" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42945 } } peers { permanent_uuid: "e19b197b894d43369530f08dabe843db" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 39937 } attrs { promote: false } } peers { permanent_uuid: "a945a1b3df7b4b8684f9b198beca1c89" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 38525 } attrs { promote: false } } }
I20260926 01:00:03.155149 10717 sys_catalog.cc:455] T 00000000000000000000000000000000 P d53706d9bbc34ab49aced7cbb4117f20 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d53706d9bbc34ab49aced7cbb4117f20" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d53706d9bbc34ab49aced7cbb4117f20" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42945 } } peers { permanent_uuid: "e19b197b894d43369530f08dabe843db" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 39937 } attrs { promote: false } } peers { permanent_uuid: "a945a1b3df7b4b8684f9b198beca1c89" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 38525 } attrs { promote: false } } }
I20260926 01:00:03.155222 10717 sys_catalog.cc:458] T 00000000000000000000000000000000 P d53706d9bbc34ab49aced7cbb4117f20 [sys.catalog]: This master's current role is: LEADER
I20260926 01:00:03.155269 10717 sys_catalog.cc:455] T 00000000000000000000000000000000 P d53706d9bbc34ab49aced7cbb4117f20 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d53706d9bbc34ab49aced7cbb4117f20" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d53706d9bbc34ab49aced7cbb4117f20" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42945 } } peers { permanent_uuid: "e19b197b894d43369530f08dabe843db" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 39937 } attrs { promote: false } } peers { permanent_uuid: "a945a1b3df7b4b8684f9b198beca1c89" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 38525 } attrs { promote: false } } }
I20260926 01:00:03.155328 10717 sys_catalog.cc:458] T 00000000000000000000000000000000 P d53706d9bbc34ab49aced7cbb4117f20 [sys.catalog]: This master's current role is: LEADER
I20260926 01:00:03.155289 10284 sys_catalog.cc:455] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d53706d9bbc34ab49aced7cbb4117f20" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d53706d9bbc34ab49aced7cbb4117f20" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42945 } } peers { permanent_uuid: "e19b197b894d43369530f08dabe843db" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 39937 } attrs { promote: false } } peers { permanent_uuid: "a945a1b3df7b4b8684f9b198beca1c89" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 38525 } attrs { promote: false } } }
I20260926 01:00:03.155367 10284 sys_catalog.cc:458] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:00:03.155393 10717 sys_catalog.cc:455] T 00000000000000000000000000000000 P d53706d9bbc34ab49aced7cbb4117f20 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d53706d9bbc34ab49aced7cbb4117f20" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d53706d9bbc34ab49aced7cbb4117f20" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42945 } } peers { permanent_uuid: "e19b197b894d43369530f08dabe843db" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 39937 } attrs { promote: false } } peers { permanent_uuid: "a945a1b3df7b4b8684f9b198beca1c89" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 38525 } attrs { promote: false } } }
I20260926 01:00:03.155463 10717 sys_catalog.cc:458] T 00000000000000000000000000000000 P d53706d9bbc34ab49aced7cbb4117f20 [sys.catalog]: This master's current role is: LEADER
I20260926 01:00:03.155661 10725 sys_catalog.cc:455] T 00000000000000000000000000000000 P e19b197b894d43369530f08dabe843db [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d53706d9bbc34ab49aced7cbb4117f20" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d53706d9bbc34ab49aced7cbb4117f20" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42945 } } peers { permanent_uuid: "e19b197b894d43369530f08dabe843db" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 39937 } attrs { promote: false } } peers { permanent_uuid: "a945a1b3df7b4b8684f9b198beca1c89" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 38525 } attrs { promote: false } } }
I20260926 01:00:03.155787 10725 sys_catalog.cc:458] T 00000000000000000000000000000000 P e19b197b894d43369530f08dabe843db [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:00:03.160560 10728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:03.156551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41674 (local address 127.3.75.3:44197)
0926 01:00:03.157246 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:03.157250 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:03.157259 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:00:03.157459 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:03.157462 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:03.157476 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:03.157551 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:00:03.157667 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:03.158634 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:03.159676 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:03.160190 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:03.160297 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:03.160376 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:03.160452 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 01:00:03.160489 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":79,"threads_started":1}
I20260926 01:00:03.160552 10726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:03.156463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.3:44197 (local address 127.0.0.1:41674)
0926 01:00:03.156598 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:03.156610 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:03.156655 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:03.157564 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:03.157565 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:03.157567 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:00:03.157607 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:03.157608 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:03.158719 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:03.158720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:03.159587 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:03.159590 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:03.160301 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:03.160302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:03.160374 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:03.160379 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 01:00:03.160438 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 01:00:03.160477 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":96,"threads_started":1}
I20260926 01:00:03.160665 10019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:03.156347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.2:44115 (local address 127.0.0.1:35306)
0926 01:00:03.156385 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:03.156397 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:03.156432 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:03.157025 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:03.157026 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:03.157028 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:00:03.157077 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:03.157079 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:03.158507 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:03.158508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:03.159538 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:03.159539 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:03.160427 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:03.160429 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:03.160497 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:03.160502 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 01:00:03.160533 (+ 31us) client_negotiation.cc:240] Negotiation successful
0926 01:00:03.160572 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19,"mutex_wait_us":4}
I20260926 01:00:03.160666 10727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:03.156471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.1:33259 (local address 127.0.0.1:50062)
0926 01:00:03.157064 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:03.157075 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:03.157126 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:03.157293 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:03.157294 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:03.157296 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:00:03.157333 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:03.157335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:03.158272 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:03.158273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:03.159520 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:03.159523 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:03.160428 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:03.160429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:03.160497 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:03.160503 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 01:00:03.160533 (+ 30us) client_negotiation.cc:240] Negotiation successful
0926 01:00:03.160572 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"mutex_wait_us":6,"thread_start_us":106,"threads_started":1}
I20260926 01:00:03.160979 10729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:03.156698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35306 (local address 127.3.75.2:44115)
0926 01:00:03.156875 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:03.156880 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:03.156891 (+ 11us) server_negotiation.cc:407] Connection header received
0926 01:00:03.156932 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:03.156935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:03.156952 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:03.157031 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:00:03.157299 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:03.158422 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:03.159620 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:03.160328 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:03.160762 (+ 434us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:03.160837 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:03.160859 (+ 22us) server_negotiation.cc:299] Negotiation successful
0926 01:00:03.160904 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":78,"threads_started":1}
I20260926 01:00:03.161402 10730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:03.156711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50062 (local address 127.3.75.1:33259)
0926 01:00:03.156934 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:03.156939 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:03.157142 (+ 203us) server_negotiation.cc:407] Connection header received
0926 01:00:03.157193 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:03.157195 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:03.157213 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:03.157286 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:00:03.157397 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:03.158185 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:03.159620 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:03.160323 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:03.160695 (+ 372us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:03.160780 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:03.160819 (+ 39us) server_negotiation.cc:299] Negotiation successful
0926 01:00:03.160866 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":106,"threads_started":1}
I20260926 01:00:03.161568 10632 tablet_service.cc:1467] Processing CreateTablet for tablet b8b315b535f84071bacbae79e26a728a (DEFAULT_TABLE table=test-table [id=6cf148797bd343e0a8ecc1ea60f283ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:00:03.161670 10496 tablet_service.cc:1467] Processing CreateTablet for tablet b8b315b535f84071bacbae79e26a728a (DEFAULT_TABLE table=test-table [id=6cf148797bd343e0a8ecc1ea60f283ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:00:03.161866 10632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8b315b535f84071bacbae79e26a728a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:00:03.162012 10496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8b315b535f84071bacbae79e26a728a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:00:03.162165 10360 tablet_service.cc:1467] Processing CreateTablet for tablet b8b315b535f84071bacbae79e26a728a (DEFAULT_TABLE table=test-table [id=6cf148797bd343e0a8ecc1ea60f283ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:00:03.162470 10360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8b315b535f84071bacbae79e26a728a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:00:03.164741 10731 tablet_bootstrap.cc:492] T b8b315b535f84071bacbae79e26a728a P e6356827c4004b85bf0c0009f1e84df5: Bootstrap starting.
I20260926 01:00:03.165429 10731 tablet_bootstrap.cc:654] T b8b315b535f84071bacbae79e26a728a P e6356827c4004b85bf0c0009f1e84df5: Neither blocks nor log segments found. Creating new log.
I20260926 01:00:03.165426 10732 tablet_bootstrap.cc:492] T b8b315b535f84071bacbae79e26a728a P 17a6cffb4a6a4ab3bc6f18c12a63b38e: Bootstrap starting.
I20260926 01:00:03.165709 10733 tablet_bootstrap.cc:492] T b8b315b535f84071bacbae79e26a728a P c7ac97c190134edfbd953cc9e2501843: Bootstrap starting.
I20260926 01:00:03.165895 10731 log.cc:824] T b8b315b535f84071bacbae79e26a728a P e6356827c4004b85bf0c0009f1e84df5: Log is configured to *not* fsync() on all Append() calls
I20260926 01:00:03.166054 10732 tablet_bootstrap.cc:654] T b8b315b535f84071bacbae79e26a728a P 17a6cffb4a6a4ab3bc6f18c12a63b38e: Neither blocks nor log segments found. Creating new log.
I20260926 01:00:03.166373 10732 log.cc:824] T b8b315b535f84071bacbae79e26a728a P 17a6cffb4a6a4ab3bc6f18c12a63b38e: Log is configured to *not* fsync() on all Append() calls
I20260926 01:00:03.166493 10733 tablet_bootstrap.cc:654] T b8b315b535f84071bacbae79e26a728a P c7ac97c190134edfbd953cc9e2501843: Neither blocks nor log segments found. Creating new log.
I20260926 01:00:03.166728 10731 tablet_bootstrap.cc:492] T b8b315b535f84071bacbae79e26a728a P e6356827c4004b85bf0c0009f1e84df5: No bootstrap required, opened a new log
I20260926 01:00:03.166788 10733 log.cc:824] T b8b315b535f84071bacbae79e26a728a P c7ac97c190134edfbd953cc9e2501843: Log is configured to *not* fsync() on all Append() calls
I20260926 01:00:03.166813 10731 ts_tablet_manager.cc:1397] T b8b315b535f84071bacbae79e26a728a P e6356827c4004b85bf0c0009f1e84df5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:00:03.167099 10732 tablet_bootstrap.cc:492] T b8b315b535f84071bacbae79e26a728a P 17a6cffb4a6a4ab3bc6f18c12a63b38e: No bootstrap required, opened a new log
I20260926 01:00:03.167172 10732 ts_tablet_manager.cc:1397] T b8b315b535f84071bacbae79e26a728a P 17a6cffb4a6a4ab3bc6f18c12a63b38e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:00:03.167652 10733 tablet_bootstrap.cc:492] T b8b315b535f84071bacbae79e26a728a P c7ac97c190134edfbd953cc9e2501843: No bootstrap required, opened a new log
I20260926 01:00:03.167721 10733 ts_tablet_manager.cc:1397] T b8b315b535f84071bacbae79e26a728a P c7ac97c190134edfbd953cc9e2501843: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:00:03.169296 10731 raft_consensus.cc:348] T b8b315b535f84071bacbae79e26a728a P e6356827c4004b85bf0c0009f1e84df5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6356827c4004b85bf0c0009f1e84df5" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 44115 } } peers { permanent_uuid: "17a6cffb4a6a4ab3bc6f18c12a63b38e" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 44197 } } peers { permanent_uuid: "c7ac97c190134edfbd953cc9e2501843" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 33259 } }
I20260926 01:00:03.169449 10731 raft_consensus.cc:374] T b8b315b535f84071bacbae79e26a728a P e6356827c4004b85bf0c0009f1e84df5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:00:03.169337 10732 raft_consensus.cc:348] T b8b315b535f84071bacbae79e26a728a P 17a6cffb4a6a4ab3bc6f18c12a63b38e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6356827c4004b85bf0c0009f1e84df5" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 44115 } } peers { permanent_uuid: "17a6cffb4a6a4ab3bc6f18c12a63b38e" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 44197 } } peers { permanent_uuid: "c7ac97c190134edfbd953cc9e2501843" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 33259 } }
I20260926 01:00:03.169481 10731 raft_consensus.cc:729] T b8b315b535f84071bacbae79e26a728a P e6356827c4004b85bf0c0009f1e84df5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6356827c4004b85bf0c0009f1e84df5, State: Initialized, Role: FOLLOWER
I20260926 01:00:03.169517 10732 raft_consensus.cc:374] T b8b315b535f84071bacbae79e26a728a P 17a6cffb4a6a4ab3bc6f18c12a63b38e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:00:03.169571 10732 raft_consensus.cc:729] T b8b315b535f84071bacbae79e26a728a P 17a6cffb4a6a4ab3bc6f18c12a63b38e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17a6cffb4a6a4ab3bc6f18c12a63b38e, State: Initialized, Role: FOLLOWER
I20260926 01:00:03.169661 10731 consensus_queue.cc:260] T b8b315b535f84071bacbae79e26a728a P e6356827c4004b85bf0c0009f1e84df5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6356827c4004b85bf0c0009f1e84df5" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 44115 } } peers { permanent_uuid: "17a6cffb4a6a4ab3bc6f18c12a63b38e" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 44197 } } peers { permanent_uuid: "c7ac97c190134edfbd953cc9e2501843" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 33259 } }
I20260926 01:00:03.169677 10732 consensus_queue.cc:260] T b8b315b535f84071bacbae79e26a728a P 17a6cffb4a6a4ab3bc6f18c12a63b38e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6356827c4004b85bf0c0009f1e84df5" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 44115 } } peers { permanent_uuid: "17a6cffb4a6a4ab3bc6f18c12a63b38e" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 44197 } } peers { permanent_uuid: "c7ac97c190134edfbd953cc9e2501843" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 33259 } }
I20260926 01:00:03.169713 10733 raft_consensus.cc:348] T b8b315b535f84071bacbae79e26a728a P c7ac97c190134edfbd953cc9e2501843 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6356827c4004b85bf0c0009f1e84df5" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 44115 } } peers { permanent_uuid: "17a6cffb4a6a4ab3bc6f18c12a63b38e" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 44197 } } peers { permanent_uuid: "c7ac97c190134edfbd953cc9e2501843" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 33259 } }
I20260926 01:00:03.169855 10733 raft_consensus.cc:374] T b8b315b535f84071bacbae79e26a728a P c7ac97c190134edfbd953cc9e2501843 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:00:03.169898 10733 raft_consensus.cc:729] T b8b315b535f84071bacbae79e26a728a P c7ac97c190134edfbd953cc9e2501843 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7ac97c190134edfbd953cc9e2501843, State: Initialized, Role: FOLLOWER
I20260926 01:00:03.169916 10732 ts_tablet_manager.cc:1428] T b8b315b535f84071bacbae79e26a728a P 17a6cffb4a6a4ab3bc6f18c12a63b38e: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 01:00:03.169924 10731 ts_tablet_manager.cc:1428] T b8b315b535f84071bacbae79e26a728a P e6356827c4004b85bf0c0009f1e84df5: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 01:00:03.170037 10562 heartbeater.cc:503] Master 127.3.75.62:42945 was elected leader, sending a full tablet report...
I20260926 01:00:03.170017 10733 consensus_queue.cc:260] T b8b315b535f84071bacbae79e26a728a P c7ac97c190134edfbd953cc9e2501843 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6356827c4004b85bf0c0009f1e84df5" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 44115 } } peers { permanent_uuid: "17a6cffb4a6a4ab3bc6f18c12a63b38e" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 44197 } } peers { permanent_uuid: "c7ac97c190134edfbd953cc9e2501843" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 33259 } }
I20260926 01:00:03.170219 10733 ts_tablet_manager.cc:1428] T b8b315b535f84071bacbae79e26a728a P c7ac97c190134edfbd953cc9e2501843: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 01:00:03.170367 10426 heartbeater.cc:503] Master 127.3.75.62:42945 was elected leader, sending a full tablet report...
I20260926 01:00:03.170862 10698 heartbeater.cc:503] Master 127.3.75.62:42945 was elected leader, sending a full tablet report...
I20260926 01:00:03.175419 10739 raft_consensus.cc:482] T b8b315b535f84071bacbae79e26a728a P c7ac97c190134edfbd953cc9e2501843 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:00:03.175506 10739 raft_consensus.cc:504] T b8b315b535f84071bacbae79e26a728a P c7ac97c190134edfbd953cc9e2501843 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6356827c4004b85bf0c0009f1e84df5" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 44115 } } peers { permanent_uuid: "17a6cffb4a6a4ab3bc6f18c12a63b38e" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 44197 } } peers { permanent_uuid: "c7ac97c190134edfbd953cc9e2501843" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 33259 } }
I20260926 01:00:03.175803 10739 leader_election.cc:288] T b8b315b535f84071bacbae79e26a728a P c7ac97c190134edfbd953cc9e2501843 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6356827c4004b85bf0c0009f1e84df5 (127.3.75.2:44115), 17a6cffb4a6a4ab3bc6f18c12a63b38e (127.3.75.3:44197)
I20260926 01:00:03.178557 10729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:03.176009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:48971 (local address 127.3.75.2:44115)
0926 01:00:03.176048 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:03.176050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:03.176059 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:00:03.176097 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:03.176097 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:03.176106 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:03.176157 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:00:03.176300 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:03.176867 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:03.177799 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:03.178216 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:03.178292 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:03.178301 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:03.178463 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 01:00:03.178495 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 01:00:03.178561 10430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:03.175919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.2:44115 (local address 127.3.75.1:48971)
0926 01:00:03.175973 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:03.175984 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:03.176022 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:03.176164 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:03.176165 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:03.176166 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:00:03.176232 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:03.176234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:03.176940 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:03.176941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:03.177702 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:03.177704 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:03.178298 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:03.178299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:03.178378 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:03.178381 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:00:03.178456 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 01:00:03.178496 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":25}
I20260926 01:00:03.178614 10728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:03.176236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:59223 (local address 127.3.75.3:44197)
0926 01:00:03.176287 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:03.176288 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:03.176295 (+ 7us) server_negotiation.cc:407] Connection header received
0926 01:00:03.176314 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:03.176314 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:03.176323 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:03.176398 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:00:03.176535 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:03.176994 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:03.177885 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:03.178256 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:03.178349 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:03.178358 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:03.178520 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 01:00:03.178552 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 01:00:03.178663 10431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:03.176039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.3:44197 (local address 127.3.75.1:59223)
0926 01:00:03.176072 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:03.176082 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:03.176117 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:03.176401 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:03.176402 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:03.176404 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:00:03.176461 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:03.176463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:03.177096 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:03.177097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:03.177799 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:03.177800 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:03.178353 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:03.178353 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:03.178429 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:03.178438 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 01:00:03.178514 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 01:00:03.178552 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260926 01:00:03.178961 10652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8b315b535f84071bacbae79e26a728a" candidate_uuid: "c7ac97c190134edfbd953cc9e2501843" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17a6cffb4a6a4ab3bc6f18c12a63b38e" is_pre_election: true
I20260926 01:00:03.179016 10516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8b315b535f84071bacbae79e26a728a" candidate_uuid: "c7ac97c190134edfbd953cc9e2501843" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6356827c4004b85bf0c0009f1e84df5" is_pre_election: true
I20260926 01:00:03.179087 10652 raft_consensus.cc:2454] T b8b315b535f84071bacbae79e26a728a P 17a6cffb4a6a4ab3bc6f18c12a63b38e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7ac97c190134edfbd953cc9e2501843 in term 0.
I20260926 01:00:03.179123 10516 raft_consensus.cc:2454] T b8b315b535f84071bacbae79e26a728a P e6356827c4004b85bf0c0009f1e84df5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7ac97c190134edfbd953cc9e2501843 in term 0.
I20260926 01:00:03.179288 10313 leader_election.cc:302] T b8b315b535f84071bacbae79e26a728a P c7ac97c190134edfbd953cc9e2501843 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17a6cffb4a6a4ab3bc6f18c12a63b38e, c7ac97c190134edfbd953cc9e2501843; no voters:
I20260926 01:00:03.179409 10739 raft_consensus.cc:2781] T b8b315b535f84071bacbae79e26a728a P c7ac97c190134edfbd953cc9e2501843 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:00:03.179459 10739 raft_consensus.cc:482] T b8b315b535f84071bacbae79e26a728a P c7ac97c190134edfbd953cc9e2501843 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:00:03.179489 10739 raft_consensus.cc:3037] T b8b315b535f84071bacbae79e26a728a P c7ac97c190134edfbd953cc9e2501843 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:00:03.180058 10739 raft_consensus.cc:504] T b8b315b535f84071bacbae79e26a728a P c7ac97c190134edfbd953cc9e2501843 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6356827c4004b85bf0c0009f1e84df5" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 44115 } } peers { permanent_uuid: "17a6cffb4a6a4ab3bc6f18c12a63b38e" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 44197 } } peers { permanent_uuid: "c7ac97c190134edfbd953cc9e2501843" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 33259 } }
I20260926 01:00:03.180181 10739 leader_election.cc:288] T b8b315b535f84071bacbae79e26a728a P c7ac97c190134edfbd953cc9e2501843 [CANDIDATE]: Term 1 election: Requested vote from peers e6356827c4004b85bf0c0009f1e84df5 (127.3.75.2:44115), 17a6cffb4a6a4ab3bc6f18c12a63b38e (127.3.75.3:44197)
I20260926 01:00:03.180322 10516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8b315b535f84071bacbae79e26a728a" candidate_uuid: "c7ac97c190134edfbd953cc9e2501843" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6356827c4004b85bf0c0009f1e84df5"
I20260926 01:00:03.180346 10652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8b315b535f84071bacbae79e26a728a" candidate_uuid: "c7ac97c190134edfbd953cc9e2501843" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17a6cffb4a6a4ab3bc6f18c12a63b38e"
I20260926 01:00:03.180395 10516 raft_consensus.cc:3037] T b8b315b535f84071bacbae79e26a728a P e6356827c4004b85bf0c0009f1e84df5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:00:03.180445 10652 raft_consensus.cc:3037] T b8b315b535f84071bacbae79e26a728a P 17a6cffb4a6a4ab3bc6f18c12a63b38e [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:00:03.181164 10516 raft_consensus.cc:2454] T b8b315b535f84071bacbae79e26a728a P e6356827c4004b85bf0c0009f1e84df5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7ac97c190134edfbd953cc9e2501843 in term 1.
I20260926 01:00:03.181228 10652 raft_consensus.cc:2454] T b8b315b535f84071bacbae79e26a728a P 17a6cffb4a6a4ab3bc6f18c12a63b38e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7ac97c190134edfbd953cc9e2501843 in term 1.
I20260926 01:00:03.181357 10313 leader_election.cc:302] T b8b315b535f84071bacbae79e26a728a P c7ac97c190134edfbd953cc9e2501843 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c7ac97c190134edfbd953cc9e2501843, e6356827c4004b85bf0c0009f1e84df5; no voters:
I20260926 01:00:03.181453 10739 raft_consensus.cc:2781] T b8b315b535f84071bacbae79e26a728a P c7ac97c190134edfbd953cc9e2501843 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:00:03.181591 10739 raft_consensus.cc:686] T b8b315b535f84071bacbae79e26a728a P c7ac97c190134edfbd953cc9e2501843 [term 1 LEADER]: Becoming Leader. State: Replica: c7ac97c190134edfbd953cc9e2501843, State: Running, Role: LEADER
I20260926 01:00:03.181699 10739 consensus_queue.cc:237] T b8b315b535f84071bacbae79e26a728a P c7ac97c190134edfbd953cc9e2501843 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6356827c4004b85bf0c0009f1e84df5" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 44115 } } peers { permanent_uuid: "17a6cffb4a6a4ab3bc6f18c12a63b38e" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 44197 } } peers { permanent_uuid: "c7ac97c190134edfbd953cc9e2501843" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 33259 } }
I20260926 01:00:03.182464 9462 catalog_manager.cc:5495] T b8b315b535f84071bacbae79e26a728a P c7ac97c190134edfbd953cc9e2501843 reported cstate change: term changed from 0 to 1, leader changed from <none> to c7ac97c190134edfbd953cc9e2501843 (127.3.75.1). New cstate: current_term: 1 leader_uuid: "c7ac97c190134edfbd953cc9e2501843" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6356827c4004b85bf0c0009f1e84df5" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 44115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17a6cffb4a6a4ab3bc6f18c12a63b38e" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 44197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c7ac97c190134edfbd953cc9e2501843" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 33259 } health_report { overall_health: HEALTHY } } }
I20260926 01:00:03.193835 10730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:03.191180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50064 (local address 127.3.75.1:33259)
0926 01:00:03.191241 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:03.191244 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:03.191252 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:00:03.191318 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:03.191319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:03.191336 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:03.191405 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:00:03.191533 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:03.192047 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:03.192894 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:03.193150 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:03.193216 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:03.193409 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:00:03.193522 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:00:03.193725 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:03.193748 (+ 23us) server_negotiation.cc:299] Negotiation successful
0926 01:00:03.193767 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 01:00:03.195526 10652 raft_consensus.cc:1261] T b8b315b535f84071bacbae79e26a728a P 17a6cffb4a6a4ab3bc6f18c12a63b38e [term 1 FOLLOWER]: Refusing update from remote peer c7ac97c190134edfbd953cc9e2501843: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:00:03.195526 10516 raft_consensus.cc:1261] T b8b315b535f84071bacbae79e26a728a P e6356827c4004b85bf0c0009f1e84df5 [term 1 FOLLOWER]: Refusing update from remote peer c7ac97c190134edfbd953cc9e2501843: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:00:03.195816 10739 consensus_queue.cc:1035] T b8b315b535f84071bacbae79e26a728a P c7ac97c190134edfbd953cc9e2501843 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17a6cffb4a6a4ab3bc6f18c12a63b38e" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 44197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:00:03.195930 10740 consensus_queue.cc:1035] T b8b315b535f84071bacbae79e26a728a P c7ac97c190134edfbd953cc9e2501843 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6356827c4004b85bf0c0009f1e84df5" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 44115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:00:03.199308 10744 mvcc.cc:205] Tried to move back new op lower bound from 7333414515486339072 to 7333414515432947712. Current Snapshot: MvccSnapshot[applied={T|T < 7333414515486339072 or (T in {7333414515486339072})}]
I20260926 01:00:03.200649 10745 mvcc.cc:205] Tried to move back new op lower bound from 7333414515486339072 to 7333414515432947712. Current Snapshot: MvccSnapshot[applied={T|T < 7333414515486339072}]
I20260926 01:00:03.201519 9462 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:00:03.201604 9462 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:00:03.203040 9462 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d53706d9bbc34ab49aced7cbb4117f20: Sending DeleteTablet for 3 replicas of tablet b8b315b535f84071bacbae79e26a728a
I20260926 01:00:03.203334 10496 tablet_service.cc:1514] Processing DeleteTablet for tablet b8b315b535f84071bacbae79e26a728a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:00:03 UTC) from {username='oryx', principal='oryx/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:35306
I20260926 01:00:03.203368 10632 tablet_service.cc:1514] Processing DeleteTablet for tablet b8b315b535f84071bacbae79e26a728a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:00:03 UTC) from {username='oryx', principal='oryx/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:41674
I20260926 01:00:03.203555 10360 tablet_service.cc:1514] Processing DeleteTablet for tablet b8b315b535f84071bacbae79e26a728a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:00:03 UTC) from {username='oryx', principal='oryx/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:50062
I20260926 01:00:03.203763 10754 tablet_replica.cc:330] T b8b315b535f84071bacbae79e26a728a P c7ac97c190134edfbd953cc9e2501843: stopping tablet replica
I20260926 01:00:03.203758 10752 tablet_replica.cc:330] T b8b315b535f84071bacbae79e26a728a P e6356827c4004b85bf0c0009f1e84df5: stopping tablet replica
I20260926 01:00:03.203882 10752 raft_consensus.cc:2229] T b8b315b535f84071bacbae79e26a728a P e6356827c4004b85bf0c0009f1e84df5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:00:03.203912 10754 raft_consensus.cc:2229] T b8b315b535f84071bacbae79e26a728a P c7ac97c190134edfbd953cc9e2501843 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:00:03.203987 10752 raft_consensus.cc:2258] T b8b315b535f84071bacbae79e26a728a P e6356827c4004b85bf0c0009f1e84df5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:00:03.204030 10754 raft_consensus.cc:2258] T b8b315b535f84071bacbae79e26a728a P c7ac97c190134edfbd953cc9e2501843 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:00:03.204362 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 9430
I20260926 01:00:03.205015 10754 ts_tablet_manager.cc:1905] T b8b315b535f84071bacbae79e26a728a P c7ac97c190134edfbd953cc9e2501843: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:00:03.205614 10752 ts_tablet_manager.cc:1905] T b8b315b535f84071bacbae79e26a728a P e6356827c4004b85bf0c0009f1e84df5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:00:03.206961 10754 ts_tablet_manager.cc:1918] T b8b315b535f84071bacbae79e26a728a P c7ac97c190134edfbd953cc9e2501843: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:00:03.207024 10754 log.cc:1196] T b8b315b535f84071bacbae79e26a728a P c7ac97c190134edfbd953cc9e2501843: Deleting WAL directory at /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-0/wal/wals/b8b315b535f84071bacbae79e26a728a
I20260926 01:00:03.207206 10752 ts_tablet_manager.cc:1918] T b8b315b535f84071bacbae79e26a728a P e6356827c4004b85bf0c0009f1e84df5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:00:03.207294 10754 ts_tablet_manager.cc:1939] T b8b315b535f84071bacbae79e26a728a P c7ac97c190134edfbd953cc9e2501843: Deleting consensus metadata
I20260926 01:00:03.207310 10752 log.cc:1196] T b8b315b535f84071bacbae79e26a728a P e6356827c4004b85bf0c0009f1e84df5: Deleting WAL directory at /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-1/wal/wals/b8b315b535f84071bacbae79e26a728a
I20260926 01:00:03.207568 10752 ts_tablet_manager.cc:1939] T b8b315b535f84071bacbae79e26a728a P e6356827c4004b85bf0c0009f1e84df5: Deleting consensus metadata
W20260926 01:00:03.209949 10313 connection.cc:535] server connection from 127.0.0.1:50062 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 01:00:03.210480 10452 connection.cc:535] server connection from 127.0.0.1:35306 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 01:00:03.219002 10753 tablet_replica.cc:330] T b8b315b535f84071bacbae79e26a728a P 17a6cffb4a6a4ab3bc6f18c12a63b38e: stopping tablet replica
I20260926 01:00:03.219143 10753 raft_consensus.cc:2229] T b8b315b535f84071bacbae79e26a728a P 17a6cffb4a6a4ab3bc6f18c12a63b38e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:00:03.219274 10296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:03.211602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35440 (local address 127.3.75.60:38525)
0926 01:00:03.211646 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:03.211649 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:03.211775 (+ 126us) server_negotiation.cc:407] Connection header received
0926 01:00:03.211914 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:03.211916 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:03.211927 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:03.211979 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:03.212695 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:03.213396 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:03.214107 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:03.214323 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:03.214404 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:03.216421 (+ 2017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:03.216423 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:03.216423 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:00:03.216430 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:03.218030 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:03.218404 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:03.218405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:03.218405 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:03.218459 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:03.218666 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:03.218667 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:03.218667 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:03.218806 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:03.218921 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:03.219122 (+ 201us) server_negotiation.cc:299] Negotiation successful
0926 01:00:03.219195 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 01:00:03.219283 10753 raft_consensus.cc:2258] T b8b315b535f84071bacbae79e26a728a P 17a6cffb4a6a4ab3bc6f18c12a63b38e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:00:03.219738 10753 ts_tablet_manager.cc:1905] T b8b315b535f84071bacbae79e26a728a P 17a6cffb4a6a4ab3bc6f18c12a63b38e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:00:03.220068 10713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:03.211827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41676 (local address 127.3.75.61:39937)
0926 01:00:03.211870 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:03.211871 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:03.211878 (+ 7us) server_negotiation.cc:407] Connection header received
0926 01:00:03.211900 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:03.211901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:03.211911 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:03.211963 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:03.212523 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:03.213293 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:03.214368 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:03.214661 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:03.214733 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:03.216407 (+ 1674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:03.216409 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:03.216409 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:00:03.216416 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:03.218336 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:03.218729 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:03.218730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:03.218730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:03.218781 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:03.219021 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:03.219022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:03.219022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:03.219152 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:03.219225 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:03.219882 (+ 657us) server_negotiation.cc:299] Negotiation successful
0926 01:00:03.219973 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 01:00:03.221369 10753 ts_tablet_manager.cc:1918] T b8b315b535f84071bacbae79e26a728a P 17a6cffb4a6a4ab3bc6f18c12a63b38e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:00:03.221431 10753 log.cc:1196] T b8b315b535f84071bacbae79e26a728a P 17a6cffb4a6a4ab3bc6f18c12a63b38e: Deleting WAL directory at /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-2/wal/wals/b8b315b535f84071bacbae79e26a728a
I20260926 01:00:03.221686 10753 ts_tablet_manager.cc:1939] T b8b315b535f84071bacbae79e26a728a P 17a6cffb4a6a4ab3bc6f18c12a63b38e: Deleting consensus metadata
W20260926 01:00:03.222033 10586 connection.cc:441] server connection from 127.0.0.1:41674 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41674 (request call id 1) could send its response
I20260926 01:00:04.200085 10762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:04.199787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:42945 (local address 127.3.75.3:58953)
0926 01:00:04.199940 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:04.200031 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:42945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":74,"threads_started":1}
W20260926 01:00:04.200387 10698 heartbeater.cc:644] Failed to heartbeat to 127.3.75.62:42945 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.75.62:42945: connect: Connection refused (error 111)
I20260926 01:00:04.200803 10763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:04.200388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:42945 (local address 127.3.75.1:42171)
0926 01:00:04.200681 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:04.200750 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:42945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":79,"threads_started":1}
I20260926 01:00:04.200928 10764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:04.200662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:42945 (local address 127.3.75.2:46011)
0926 01:00:04.200771 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:04.200849 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:42945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":48,"threads_started":1}
W20260926 01:00:04.201103 10562 heartbeater.cc:644] Failed to heartbeat to 127.3.75.62:42945 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.75.62:42945: connect: Connection refused (error 111)
W20260926 01:00:04.201156 10426 heartbeater.cc:644] Failed to heartbeat to 127.3.75.62:42945 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.75.62:42945: connect: Connection refused (error 111)
I20260926 01:00:04.702932 10765 raft_consensus.cc:482] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d53706d9bbc34ab49aced7cbb4117f20)
I20260926 01:00:04.703065 10765 raft_consensus.cc:504] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d53706d9bbc34ab49aced7cbb4117f20" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42945 } } peers { permanent_uuid: "e19b197b894d43369530f08dabe843db" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 39937 } attrs { promote: false } } peers { permanent_uuid: "a945a1b3df7b4b8684f9b198beca1c89" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 38525 } attrs { promote: false } }
I20260926 01:00:04.703226 10766 raft_consensus.cc:482] T 00000000000000000000000000000000 P e19b197b894d43369530f08dabe843db [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d53706d9bbc34ab49aced7cbb4117f20)
I20260926 01:00:04.703358 10765 leader_election.cc:288] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d53706d9bbc34ab49aced7cbb4117f20 (127.3.75.62:42945), e19b197b894d43369530f08dabe843db (127.3.75.61:39937)
I20260926 01:00:04.703667 10767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:04.703467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:42945 (local address 127.0.0.1:34278)
0926 01:00:04.703576 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:04.703618 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:42945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":64,"threads_started":1}
W20260926 01:00:04.703805 10229 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.75.62:42945: connect: Connection refused (error 111)
I20260926 01:00:04.704005 10766 raft_consensus.cc:504] T 00000000000000000000000000000000 P e19b197b894d43369530f08dabe843db [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d53706d9bbc34ab49aced7cbb4117f20" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42945 } } peers { permanent_uuid: "e19b197b894d43369530f08dabe843db" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 39937 } attrs { promote: false } } peers { permanent_uuid: "a945a1b3df7b4b8684f9b198beca1c89" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 38525 } attrs { promote: false } }
I20260926 01:00:04.704265 10767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:04.704149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:42945 (local address 127.0.0.1:34280)
0926 01:00:04.704208 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:04.704238 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:42945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 01:00:04.704458 10766 leader_election.cc:288] T 00000000000000000000000000000000 P e19b197b894d43369530f08dabe843db [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d53706d9bbc34ab49aced7cbb4117f20 (127.3.75.62:42945), a945a1b3df7b4b8684f9b198beca1c89 (127.3.75.60:38525)
W20260926 01:00:04.704521 10229 leader_election.cc:334] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d53706d9bbc34ab49aced7cbb4117f20 (127.3.75.62:42945): Network error: Client connection negotiation failed: client connection to 127.3.75.62:42945: connect: Connection refused (error 111)
I20260926 01:00:04.704604 10772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:04.704354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:42945 (local address 127.0.0.1:34294)
0926 01:00:04.704538 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:04.704576 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:42945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":42,"threads_started":1}
W20260926 01:00:04.704723 10043 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.75.62:42945: connect: Connection refused (error 111)
I20260926 01:00:04.705120 10772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:04.705041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:42945 (local address 127.0.0.1:34308)
0926 01:00:04.705077 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:04.705098 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:42945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260926 01:00:04.705199 10043 leader_election.cc:334] T 00000000000000000000000000000000 P e19b197b894d43369530f08dabe843db [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d53706d9bbc34ab49aced7cbb4117f20 (127.3.75.62:42945): Network error: Client connection negotiation failed: client connection to 127.3.75.62:42945: connect: Connection refused (error 111)
I20260926 01:00:04.706725 10769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:04.703704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41684 (local address 127.3.75.61:39937)
0926 01:00:04.703958 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:04.703960 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:04.703968 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:00:04.703993 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:04.703993 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:04.704005 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:04.704094 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:00:04.704346 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:04.705120 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:04.706103 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:04.706451 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:04.706502 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:04.706511 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:04.706663 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 01:00:04.706685 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":68,"threads_started":1}
I20260926 01:00:04.706790 10768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:04.703630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.61:39937 (local address 127.0.0.1:41684)
0926 01:00:04.703792 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:04.703803 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:04.703859 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:04.704122 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:04.704123 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:04.704125 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:00:04.704187 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:04.704189 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:04.705219 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:04.705220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:04.706020 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:04.706022 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:04.706527 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:04.706528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:04.706592 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:04.706596 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:00:04.706660 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 01:00:04.706698 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"thread_start_us":46,"threads_started":1}
I20260926 01:00:04.706993 10069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a945a1b3df7b4b8684f9b198beca1c89" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e19b197b894d43369530f08dabe843db" is_pre_election: true
I20260926 01:00:04.707085 10069 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e19b197b894d43369530f08dabe843db [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a945a1b3df7b4b8684f9b198beca1c89 in term 1.
I20260926 01:00:04.707279 10229 leader_election.cc:302] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a945a1b3df7b4b8684f9b198beca1c89, e19b197b894d43369530f08dabe843db; no voters: d53706d9bbc34ab49aced7cbb4117f20
I20260926 01:00:04.707378 10765 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 01:00:04.707418 10765 raft_consensus.cc:482] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d53706d9bbc34ab49aced7cbb4117f20)
I20260926 01:00:04.707445 10765 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89 [term 1 FOLLOWER]: Advancing to term 2
I20260926 01:00:04.707839 10771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:04.704385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.60:38525 (local address 127.0.0.1:35446)
0926 01:00:04.704956 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:04.704967 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:04.705030 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:04.705175 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:04.705176 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:04.705178 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:00:04.705270 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:04.705272 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:04.705891 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:04.705892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:04.706732 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:04.706735 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:04.707615 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:04.707615 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:04.707657 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:04.707660 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:00:04.707723 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 01:00:04.707765 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":47,"threads_started":1}
I20260926 01:00:04.707841 10773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:04.704548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35446 (local address 127.3.75.60:38525)
0926 01:00:04.704800 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:04.704802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:04.705034 (+ 232us) server_negotiation.cc:407] Connection header received
0926 01:00:04.705091 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:04.705092 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:04.705102 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:04.705213 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:00:04.705333 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:04.705811 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:04.706825 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:04.707515 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:04.707560 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:04.707568 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:04.707726 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 01:00:04.707787 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":73,"threads_started":1}
I20260926 01:00:04.708024 10255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e19b197b894d43369530f08dabe843db" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a945a1b3df7b4b8684f9b198beca1c89" is_pre_election: true
I20260926 01:00:04.708287 10765 raft_consensus.cc:504] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d53706d9bbc34ab49aced7cbb4117f20" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42945 } } peers { permanent_uuid: "e19b197b894d43369530f08dabe843db" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 39937 } attrs { promote: false } } peers { permanent_uuid: "a945a1b3df7b4b8684f9b198beca1c89" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 38525 } attrs { promote: false } }
I20260926 01:00:04.708408 10255 raft_consensus.cc:2379] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e19b197b894d43369530f08dabe843db in current term 2: Already voted for candidate a945a1b3df7b4b8684f9b198beca1c89 in this term.
I20260926 01:00:04.708443 10765 leader_election.cc:288] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89 [CANDIDATE]: Term 2 election: Requested vote from peers d53706d9bbc34ab49aced7cbb4117f20 (127.3.75.62:42945), e19b197b894d43369530f08dabe843db (127.3.75.61:39937)
I20260926 01:00:04.708587 10768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:04.708503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:42945 (local address 127.0.0.1:34312)
0926 01:00:04.708536 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:04.708566 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:42945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260926 01:00:04.708600 10045 leader_election.cc:302] T 00000000000000000000000000000000 P e19b197b894d43369530f08dabe843db [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e19b197b894d43369530f08dabe843db; no voters: a945a1b3df7b4b8684f9b198beca1c89, d53706d9bbc34ab49aced7cbb4117f20
I20260926 01:00:04.708724 10766 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e19b197b894d43369530f08dabe843db [term 1 FOLLOWER]: Advancing to term 2
I20260926 01:00:04.708809 10069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a945a1b3df7b4b8684f9b198beca1c89" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e19b197b894d43369530f08dabe843db"
I20260926 01:00:04.708875 10768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:04.708802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:42945 (local address 127.0.0.1:34316)
0926 01:00:04.708829 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:04.708853 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:42945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260926 01:00:04.708959 10229 leader_election.cc:334] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d53706d9bbc34ab49aced7cbb4117f20 (127.3.75.62:42945): Network error: Client connection negotiation failed: client connection to 127.3.75.62:42945: connect: Connection refused (error 111)
I20260926 01:00:04.709437 10766 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e19b197b894d43369530f08dabe843db [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 01:00:04.710103 10069 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e19b197b894d43369530f08dabe843db [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a945a1b3df7b4b8684f9b198beca1c89 in term 2.
I20260926 01:00:04.710256 10229 leader_election.cc:302] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a945a1b3df7b4b8684f9b198beca1c89, e19b197b894d43369530f08dabe843db; no voters: d53706d9bbc34ab49aced7cbb4117f20
I20260926 01:00:04.710393 10765 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 01:00:04.710526 10765 raft_consensus.cc:686] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89 [term 2 LEADER]: Becoming Leader. State: Replica: a945a1b3df7b4b8684f9b198beca1c89, State: Running, Role: LEADER
I20260926 01:00:04.710599 10765 consensus_queue.cc:237] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d53706d9bbc34ab49aced7cbb4117f20" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42945 } } peers { permanent_uuid: "e19b197b894d43369530f08dabe843db" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 39937 } attrs { promote: false } } peers { permanent_uuid: "a945a1b3df7b4b8684f9b198beca1c89" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 38525 } attrs { promote: false } }
I20260926 01:00:04.710916 10775 sys_catalog.cc:455] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a945a1b3df7b4b8684f9b198beca1c89. Latest consensus state: current_term: 2 leader_uuid: "a945a1b3df7b4b8684f9b198beca1c89" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d53706d9bbc34ab49aced7cbb4117f20" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42945 } } peers { permanent_uuid: "e19b197b894d43369530f08dabe843db" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 39937 } attrs { promote: false } } peers { permanent_uuid: "a945a1b3df7b4b8684f9b198beca1c89" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 38525 } attrs { promote: false } } }
I20260926 01:00:04.711007 10775 sys_catalog.cc:458] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89 [sys.catalog]: This master's current role is: LEADER
I20260926 01:00:04.711179 10777 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:00:04.711552 10777 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:00:04.711681 10777 catalog_manager.cc:1259] Loaded cluster ID: fd7906d0e0074f99817ab00b16016fba
I20260926 01:00:04.711719 10777 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:00:04.711831 10777 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:00:04.711881 10777 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89: Loaded TSK: 0
I20260926 01:00:04.711970 10777 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:00:04.759526 10244 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35440:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:00:04.761094 10244 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:00:04.762089 10069 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e19b197b894d43369530f08dabe843db [term 2 FOLLOWER]: Refusing update from remote peer a945a1b3df7b4b8684f9b198beca1c89: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260926 01:00:04.762111 10768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:04.761996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:42945 (local address 127.0.0.1:34348)
0926 01:00:04.762052 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:04.762085 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:42945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 01:00:04.762428 10768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:04.762330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:42945 (local address 127.0.0.1:34358)
0926 01:00:04.762376 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:04.762400 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:42945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 01:00:04.762466 10775 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e19b197b894d43369530f08dabe843db" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 39937 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260926 01:00:04.762606 10229 consensus_peers.cc:487] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89 -> Peer d53706d9bbc34ab49aced7cbb4117f20 (127.3.75.62:42945): Couldn't send request to peer d53706d9bbc34ab49aced7cbb4117f20. Status: Network error: Client connection negotiation failed: client connection to 127.3.75.62:42945: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 01:00:04.763178 10766 sys_catalog.cc:455] T 00000000000000000000000000000000 P e19b197b894d43369530f08dabe843db [sys.catalog]: SysCatalogTable state changed. Reason: New leader a945a1b3df7b4b8684f9b198beca1c89. Latest consensus state: current_term: 2 leader_uuid: "a945a1b3df7b4b8684f9b198beca1c89" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d53706d9bbc34ab49aced7cbb4117f20" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42945 } } peers { permanent_uuid: "e19b197b894d43369530f08dabe843db" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 39937 } attrs { promote: false } } peers { permanent_uuid: "a945a1b3df7b4b8684f9b198beca1c89" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 38525 } attrs { promote: false } } }
I20260926 01:00:04.763288 10766 sys_catalog.cc:458] T 00000000000000000000000000000000 P e19b197b894d43369530f08dabe843db [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:00:04.763594 10778 mvcc.cc:205] Tried to move back new op lower bound from 7333414521903620096 to 7333414521695014912. Current Snapshot: MvccSnapshot[applied={T|T < 7333414521903620096}]
I20260926 01:00:04.763684 10765 sys_catalog.cc:455] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "a945a1b3df7b4b8684f9b198beca1c89" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d53706d9bbc34ab49aced7cbb4117f20" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42945 } } peers { permanent_uuid: "e19b197b894d43369530f08dabe843db" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 39937 } attrs { promote: false } } peers { permanent_uuid: "a945a1b3df7b4b8684f9b198beca1c89" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 38525 } attrs { promote: false } } }
I20260926 01:00:04.763772 10765 sys_catalog.cc:458] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89 [sys.catalog]: This master's current role is: LEADER
I20260926 01:00:04.764089 10766 sys_catalog.cc:455] T 00000000000000000000000000000000 P e19b197b894d43369530f08dabe843db [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "a945a1b3df7b4b8684f9b198beca1c89" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d53706d9bbc34ab49aced7cbb4117f20" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42945 } } peers { permanent_uuid: "e19b197b894d43369530f08dabe843db" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 39937 } attrs { promote: false } } peers { permanent_uuid: "a945a1b3df7b4b8684f9b198beca1c89" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 38525 } attrs { promote: false } } }
I20260926 01:00:04.764061 10779 mvcc.cc:205] Tried to move back new op lower bound from 7333414521903620096 to 7333414521695014912. Current Snapshot: MvccSnapshot[applied={T|T < 7333414521903620096}]
I20260926 01:00:04.764174 10766 sys_catalog.cc:458] T 00000000000000000000000000000000 P e19b197b894d43369530f08dabe843db [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:00:04.768978 10767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:04.765555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.3:44197 (local address 127.0.0.1:41676)
0926 01:00:04.765665 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:04.765678 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:04.765733 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:04.765952 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:04.765953 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:04.765955 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:00:04.766013 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:04.766015 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:04.767123 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:04.767124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:04.767997 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:04.767999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:04.768732 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:04.768733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:04.768796 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:04.768800 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:00:04.768857 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 01:00:04.768894 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260926 01:00:04.769011 10783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:04.765646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41676 (local address 127.3.75.3:44197)
0926 01:00:04.765836 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:04.765839 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:04.765846 (+ 7us) server_negotiation.cc:407] Connection header received
0926 01:00:04.765872 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:04.765873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:04.765885 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:04.765951 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:00:04.766095 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:04.767003 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:04.768077 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:04.768651 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:04.768721 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:04.768732 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:04.768875 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 01:00:04.768913 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":62,"threads_started":1}
I20260926 01:00:04.769222 10768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:04.765647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.2:44115 (local address 127.0.0.1:35316)
0926 01:00:04.765782 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:04.765792 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:04.765831 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:04.766039 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:04.766040 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:04.766041 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:00:04.766081 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:04.766082 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:04.767001 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:04.767002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:04.768194 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:04.768210 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:04.769014 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:04.769015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:04.769080 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:04.769085 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 01:00:04.769117 (+ 32us) client_negotiation.cc:240] Negotiation successful
0926 01:00:04.769154 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":26,"thread_start_us":87,"threads_started":1}
I20260926 01:00:04.769246 10632 tablet_service.cc:1467] Processing CreateTablet for tablet 6cd604fa75514a0a8fa8561f52e4da41 (DEFAULT_TABLE table=test-table [id=629726b115724d02a67916e548714753]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:00:04.769376 10632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cd604fa75514a0a8fa8561f52e4da41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:00:04.769913 10786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:04.765832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35316 (local address 127.3.75.2:44115)
0926 01:00:04.765920 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:04.765921 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:04.765930 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:00:04.765955 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:04.765956 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:04.765966 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:04.766026 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:00:04.766140 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:04.766907 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:04.768284 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:04.768924 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:04.769768 (+ 844us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:04.769785 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:04.769807 (+ 22us) server_negotiation.cc:299] Negotiation successful
0926 01:00:04.769844 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71,"thread_start_us":42,"threads_started":1}
I20260926 01:00:04.770067 10785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:04.765576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.1:33259 (local address 127.0.0.1:50070)
0926 01:00:04.765780 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:04.765790 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:04.765832 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:04.766170 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:04.766171 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:04.766173 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:00:04.766214 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:04.766215 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:04.767186 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:04.767187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:04.769153 (+ 1966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:04.769156 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:04.769793 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:04.769794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:04.769855 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:04.769859 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:00:04.769943 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 01:00:04.769990 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186}
I20260926 01:00:04.770064 10784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:04.765673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50070 (local address 127.3.75.1:33259)
0926 01:00:04.765923 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:04.765925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:04.765933 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:00:04.765957 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:04.765958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:04.765968 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:04.766026 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:00:04.766294 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:04.767080 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:04.769229 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:04.769707 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:04.769788 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:04.769801 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:04.769951 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 01:00:04.769996 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":67,"threads_started":1}
I20260926 01:00:04.770398 10496 tablet_service.cc:1467] Processing CreateTablet for tablet 6cd604fa75514a0a8fa8561f52e4da41 (DEFAULT_TABLE table=test-table [id=629726b115724d02a67916e548714753]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:00:04.770491 10360 tablet_service.cc:1467] Processing CreateTablet for tablet 6cd604fa75514a0a8fa8561f52e4da41 (DEFAULT_TABLE table=test-table [id=629726b115724d02a67916e548714753]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:00:04.770556 10496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cd604fa75514a0a8fa8561f52e4da41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:00:04.770627 10360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cd604fa75514a0a8fa8561f52e4da41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:00:04.771060 10787 tablet_bootstrap.cc:492] T 6cd604fa75514a0a8fa8561f52e4da41 P 17a6cffb4a6a4ab3bc6f18c12a63b38e: Bootstrap starting.
I20260926 01:00:04.771422 10787 tablet_bootstrap.cc:654] T 6cd604fa75514a0a8fa8561f52e4da41 P 17a6cffb4a6a4ab3bc6f18c12a63b38e: Neither blocks nor log segments found. Creating new log.
I20260926 01:00:04.772011 10789 tablet_bootstrap.cc:492] T 6cd604fa75514a0a8fa8561f52e4da41 P e6356827c4004b85bf0c0009f1e84df5: Bootstrap starting.
I20260926 01:00:04.772056 10790 tablet_bootstrap.cc:492] T 6cd604fa75514a0a8fa8561f52e4da41 P c7ac97c190134edfbd953cc9e2501843: Bootstrap starting.
I20260926 01:00:04.772156 10787 tablet_bootstrap.cc:492] T 6cd604fa75514a0a8fa8561f52e4da41 P 17a6cffb4a6a4ab3bc6f18c12a63b38e: No bootstrap required, opened a new log
I20260926 01:00:04.772212 10787 ts_tablet_manager.cc:1397] T 6cd604fa75514a0a8fa8561f52e4da41 P 17a6cffb4a6a4ab3bc6f18c12a63b38e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 01:00:04.772356 10787 raft_consensus.cc:348] T 6cd604fa75514a0a8fa8561f52e4da41 P 17a6cffb4a6a4ab3bc6f18c12a63b38e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a6cffb4a6a4ab3bc6f18c12a63b38e" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 44197 } } peers { permanent_uuid: "e6356827c4004b85bf0c0009f1e84df5" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 44115 } } peers { permanent_uuid: "c7ac97c190134edfbd953cc9e2501843" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 33259 } }
I20260926 01:00:04.772428 10787 raft_consensus.cc:374] T 6cd604fa75514a0a8fa8561f52e4da41 P 17a6cffb4a6a4ab3bc6f18c12a63b38e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:00:04.772462 10787 raft_consensus.cc:729] T 6cd604fa75514a0a8fa8561f52e4da41 P 17a6cffb4a6a4ab3bc6f18c12a63b38e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17a6cffb4a6a4ab3bc6f18c12a63b38e, State: Initialized, Role: FOLLOWER
I20260926 01:00:04.772493 10790 tablet_bootstrap.cc:654] T 6cd604fa75514a0a8fa8561f52e4da41 P c7ac97c190134edfbd953cc9e2501843: Neither blocks nor log segments found. Creating new log.
I20260926 01:00:04.772516 10787 consensus_queue.cc:260] T 6cd604fa75514a0a8fa8561f52e4da41 P 17a6cffb4a6a4ab3bc6f18c12a63b38e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a6cffb4a6a4ab3bc6f18c12a63b38e" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 44197 } } peers { permanent_uuid: "e6356827c4004b85bf0c0009f1e84df5" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 44115 } } peers { permanent_uuid: "c7ac97c190134edfbd953cc9e2501843" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 33259 } }
I20260926 01:00:04.772687 10787 ts_tablet_manager.cc:1428] T 6cd604fa75514a0a8fa8561f52e4da41 P 17a6cffb4a6a4ab3bc6f18c12a63b38e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 01:00:04.773064 10790 tablet_bootstrap.cc:492] T 6cd604fa75514a0a8fa8561f52e4da41 P c7ac97c190134edfbd953cc9e2501843: No bootstrap required, opened a new log
I20260926 01:00:04.773120 10790 ts_tablet_manager.cc:1397] T 6cd604fa75514a0a8fa8561f52e4da41 P c7ac97c190134edfbd953cc9e2501843: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 01:00:04.773078 10789 tablet_bootstrap.cc:654] T 6cd604fa75514a0a8fa8561f52e4da41 P e6356827c4004b85bf0c0009f1e84df5: Neither blocks nor log segments found. Creating new log.
I20260926 01:00:04.773228 10790 raft_consensus.cc:348] T 6cd604fa75514a0a8fa8561f52e4da41 P c7ac97c190134edfbd953cc9e2501843 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a6cffb4a6a4ab3bc6f18c12a63b38e" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 44197 } } peers { permanent_uuid: "e6356827c4004b85bf0c0009f1e84df5" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 44115 } } peers { permanent_uuid: "c7ac97c190134edfbd953cc9e2501843" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 33259 } }
I20260926 01:00:04.773309 10790 raft_consensus.cc:374] T 6cd604fa75514a0a8fa8561f52e4da41 P c7ac97c190134edfbd953cc9e2501843 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:00:04.773373 10790 raft_consensus.cc:729] T 6cd604fa75514a0a8fa8561f52e4da41 P c7ac97c190134edfbd953cc9e2501843 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7ac97c190134edfbd953cc9e2501843, State: Initialized, Role: FOLLOWER
I20260926 01:00:04.773455 10793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:04.772871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:42945 (local address 127.3.75.3:53345)
0926 01:00:04.773375 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:04.773421 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:42945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":79,"threads_started":1}
I20260926 01:00:04.773447 10790 consensus_queue.cc:260] T 6cd604fa75514a0a8fa8561f52e4da41 P c7ac97c190134edfbd953cc9e2501843 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a6cffb4a6a4ab3bc6f18c12a63b38e" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 44197 } } peers { permanent_uuid: "e6356827c4004b85bf0c0009f1e84df5" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 44115 } } peers { permanent_uuid: "c7ac97c190134edfbd953cc9e2501843" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 33259 } }
I20260926 01:00:04.773607 10790 ts_tablet_manager.cc:1428] T 6cd604fa75514a0a8fa8561f52e4da41 P c7ac97c190134edfbd953cc9e2501843: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 01:00:04.773845 10793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:04.773671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:42945 (local address 127.3.75.3:36477)
0926 01:00:04.773795 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:04.773821 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:42945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":74,"threads_started":1}
I20260926 01:00:04.774011 10797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:04.773755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:42945 (local address 127.3.75.1:41183)
0926 01:00:04.773870 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:04.773978 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:42945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":78,"threads_started":1}
I20260926 01:00:04.774039 10427 heartbeater.cc:503] Master 127.3.75.60:38525 was elected leader, sending a full tablet report...
I20260926 01:00:04.774297 10789 tablet_bootstrap.cc:492] T 6cd604fa75514a0a8fa8561f52e4da41 P e6356827c4004b85bf0c0009f1e84df5: No bootstrap required, opened a new log
I20260926 01:00:04.774358 10789 ts_tablet_manager.cc:1397] T 6cd604fa75514a0a8fa8561f52e4da41 P e6356827c4004b85bf0c0009f1e84df5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:00:04.774392 10797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:04.774278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:42945 (local address 127.3.75.1:35563)
0926 01:00:04.774345 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:04.774370 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:42945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 01:00:04.774490 10789 raft_consensus.cc:348] T 6cd604fa75514a0a8fa8561f52e4da41 P e6356827c4004b85bf0c0009f1e84df5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a6cffb4a6a4ab3bc6f18c12a63b38e" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 44197 } } peers { permanent_uuid: "e6356827c4004b85bf0c0009f1e84df5" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 44115 } } peers { permanent_uuid: "c7ac97c190134edfbd953cc9e2501843" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 33259 } }
I20260926 01:00:04.774554 10789 raft_consensus.cc:374] T 6cd604fa75514a0a8fa8561f52e4da41 P e6356827c4004b85bf0c0009f1e84df5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:00:04.774606 10789 raft_consensus.cc:729] T 6cd604fa75514a0a8fa8561f52e4da41 P e6356827c4004b85bf0c0009f1e84df5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6356827c4004b85bf0c0009f1e84df5, State: Initialized, Role: FOLLOWER
I20260926 01:00:04.774658 10789 consensus_queue.cc:260] T 6cd604fa75514a0a8fa8561f52e4da41 P e6356827c4004b85bf0c0009f1e84df5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a6cffb4a6a4ab3bc6f18c12a63b38e" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 44197 } } peers { permanent_uuid: "e6356827c4004b85bf0c0009f1e84df5" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 44115 } } peers { permanent_uuid: "c7ac97c190134edfbd953cc9e2501843" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 33259 } }
I20260926 01:00:04.774844 10789 ts_tablet_manager.cc:1428] T 6cd604fa75514a0a8fa8561f52e4da41 P e6356827c4004b85bf0c0009f1e84df5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:04.775218 10563 heartbeater.cc:503] Master 127.3.75.60:38525 was elected leader, sending a full tablet report...
I20260926 01:00:04.775419 10799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:04.774962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:42945 (local address 127.3.75.2:54013)
0926 01:00:04.775341 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:04.775374 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:42945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":77,"threads_started":1}
I20260926 01:00:04.775730 10799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:04.775627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:42945 (local address 127.3.75.2:35219)
0926 01:00:04.775682 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:04.775709 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:42945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 01:00:04.861016 10429 tablet.cc:2369] T 6cd604fa75514a0a8fa8561f52e4da41 P c7ac97c190134edfbd953cc9e2501843: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 01:00:04.880285 10701 tablet.cc:2369] T 6cd604fa75514a0a8fa8561f52e4da41 P 17a6cffb4a6a4ab3bc6f18c12a63b38e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:00:04.898352 10798 raft_consensus.cc:482] T 6cd604fa75514a0a8fa8561f52e4da41 P e6356827c4004b85bf0c0009f1e84df5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:00:04.898432 10798 raft_consensus.cc:504] T 6cd604fa75514a0a8fa8561f52e4da41 P e6356827c4004b85bf0c0009f1e84df5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a6cffb4a6a4ab3bc6f18c12a63b38e" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 44197 } } peers { permanent_uuid: "e6356827c4004b85bf0c0009f1e84df5" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 44115 } } peers { permanent_uuid: "c7ac97c190134edfbd953cc9e2501843" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 33259 } }
I20260926 01:00:04.898803 10798 leader_election.cc:288] T 6cd604fa75514a0a8fa8561f52e4da41 P e6356827c4004b85bf0c0009f1e84df5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17a6cffb4a6a4ab3bc6f18c12a63b38e (127.3.75.3:44197), c7ac97c190134edfbd953cc9e2501843 (127.3.75.1:33259)
I20260926 01:00:04.901609 10783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:04.898999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:56977 (local address 127.3.75.3:44197)
0926 01:00:04.899088 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:04.899091 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:04.899104 (+ 13us) server_negotiation.cc:407] Connection header received
0926 01:00:04.899135 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:04.899136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:04.899149 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:04.899209 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:00:04.899353 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:04.899816 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:04.900878 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:04.901254 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:04.901345 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:04.901359 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:04.901504 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 01:00:04.901545 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 01:00:04.901609 10799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:04.898940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.3:44197 (local address 127.3.75.2:56977)
0926 01:00:04.899003 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:04.899013 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:04.899079 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:04.899213 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:04.899215 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:04.899217 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:00:04.899286 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:04.899288 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:04.899890 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:04.899891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:04.900776 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:04.900778 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:04.901348 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:04.901349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:04.901418 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:04.901422 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:00:04.901495 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 01:00:04.901535 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":52}
I20260926 01:00:04.901714 10800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:04.898996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.1:33259 (local address 127.3.75.2:39491)
0926 01:00:04.899196 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:04.899205 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:04.899249 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:04.899393 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:04.899395 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:04.899396 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:00:04.899425 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:04.899426 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:04.900115 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:04.900116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:04.900834 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:04.900837 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:04.901446 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:04.901447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:04.901507 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:04.901511 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:00:04.901571 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 01:00:04.901613 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":131,"threads_started":1}
I20260926 01:00:04.901703 10784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:04.899239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:39491 (local address 127.3.75.1:33259)
0926 01:00:04.899289 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:04.899291 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:04.899299 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:00:04.899323 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:04.899324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:04.899337 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:04.899383 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:00:04.899470 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:04.900026 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:04.900957 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:04.901340 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:04.901422 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:04.901434 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:04.901592 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 01:00:04.901636 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 01:00:04.901975 10380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd604fa75514a0a8fa8561f52e4da41" candidate_uuid: "e6356827c4004b85bf0c0009f1e84df5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7ac97c190134edfbd953cc9e2501843" is_pre_election: true
I20260926 01:00:04.902096 10652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd604fa75514a0a8fa8561f52e4da41" candidate_uuid: "e6356827c4004b85bf0c0009f1e84df5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17a6cffb4a6a4ab3bc6f18c12a63b38e" is_pre_election: true
I20260926 01:00:04.902141 10380 raft_consensus.cc:2454] T 6cd604fa75514a0a8fa8561f52e4da41 P c7ac97c190134edfbd953cc9e2501843 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6356827c4004b85bf0c0009f1e84df5 in term 0.
I20260926 01:00:04.902165 10652 raft_consensus.cc:2454] T 6cd604fa75514a0a8fa8561f52e4da41 P 17a6cffb4a6a4ab3bc6f18c12a63b38e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6356827c4004b85bf0c0009f1e84df5 in term 0.
I20260926 01:00:04.902326 10452 leader_election.cc:302] T 6cd604fa75514a0a8fa8561f52e4da41 P e6356827c4004b85bf0c0009f1e84df5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c7ac97c190134edfbd953cc9e2501843, e6356827c4004b85bf0c0009f1e84df5; no voters:
I20260926 01:00:04.902463 10798 raft_consensus.cc:2781] T 6cd604fa75514a0a8fa8561f52e4da41 P e6356827c4004b85bf0c0009f1e84df5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:00:04.902518 10798 raft_consensus.cc:482] T 6cd604fa75514a0a8fa8561f52e4da41 P e6356827c4004b85bf0c0009f1e84df5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:00:04.902550 10798 raft_consensus.cc:3037] T 6cd604fa75514a0a8fa8561f52e4da41 P e6356827c4004b85bf0c0009f1e84df5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:00:04.903290 10798 raft_consensus.cc:504] T 6cd604fa75514a0a8fa8561f52e4da41 P e6356827c4004b85bf0c0009f1e84df5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a6cffb4a6a4ab3bc6f18c12a63b38e" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 44197 } } peers { permanent_uuid: "e6356827c4004b85bf0c0009f1e84df5" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 44115 } } peers { permanent_uuid: "c7ac97c190134edfbd953cc9e2501843" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 33259 } }
I20260926 01:00:04.903437 10798 leader_election.cc:288] T 6cd604fa75514a0a8fa8561f52e4da41 P e6356827c4004b85bf0c0009f1e84df5 [CANDIDATE]: Term 1 election: Requested vote from peers 17a6cffb4a6a4ab3bc6f18c12a63b38e (127.3.75.3:44197), c7ac97c190134edfbd953cc9e2501843 (127.3.75.1:33259)
I20260926 01:00:04.903592 10380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd604fa75514a0a8fa8561f52e4da41" candidate_uuid: "e6356827c4004b85bf0c0009f1e84df5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7ac97c190134edfbd953cc9e2501843"
I20260926 01:00:04.903589 10652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd604fa75514a0a8fa8561f52e4da41" candidate_uuid: "e6356827c4004b85bf0c0009f1e84df5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17a6cffb4a6a4ab3bc6f18c12a63b38e"
I20260926 01:00:04.903659 10380 raft_consensus.cc:3037] T 6cd604fa75514a0a8fa8561f52e4da41 P c7ac97c190134edfbd953cc9e2501843 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:00:04.903659 10652 raft_consensus.cc:3037] T 6cd604fa75514a0a8fa8561f52e4da41 P 17a6cffb4a6a4ab3bc6f18c12a63b38e [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:00:04.904412 10652 raft_consensus.cc:2454] T 6cd604fa75514a0a8fa8561f52e4da41 P 17a6cffb4a6a4ab3bc6f18c12a63b38e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6356827c4004b85bf0c0009f1e84df5 in term 1.
I20260926 01:00:04.904490 10380 raft_consensus.cc:2454] T 6cd604fa75514a0a8fa8561f52e4da41 P c7ac97c190134edfbd953cc9e2501843 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6356827c4004b85bf0c0009f1e84df5 in term 1.
I20260926 01:00:04.904588 10449 leader_election.cc:302] T 6cd604fa75514a0a8fa8561f52e4da41 P e6356827c4004b85bf0c0009f1e84df5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17a6cffb4a6a4ab3bc6f18c12a63b38e, e6356827c4004b85bf0c0009f1e84df5; no voters:
I20260926 01:00:04.904681 10798 raft_consensus.cc:2781] T 6cd604fa75514a0a8fa8561f52e4da41 P e6356827c4004b85bf0c0009f1e84df5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:00:04.904807 10798 raft_consensus.cc:686] T 6cd604fa75514a0a8fa8561f52e4da41 P e6356827c4004b85bf0c0009f1e84df5 [term 1 LEADER]: Becoming Leader. State: Replica: e6356827c4004b85bf0c0009f1e84df5, State: Running, Role: LEADER
I20260926 01:00:04.904909 10798 consensus_queue.cc:237] T 6cd604fa75514a0a8fa8561f52e4da41 P e6356827c4004b85bf0c0009f1e84df5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a6cffb4a6a4ab3bc6f18c12a63b38e" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 44197 } } peers { permanent_uuid: "e6356827c4004b85bf0c0009f1e84df5" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 44115 } } peers { permanent_uuid: "c7ac97c190134edfbd953cc9e2501843" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 33259 } }
I20260926 01:00:04.905488 10800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:04.905376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:42945 (local address 127.3.75.2:58489)
0926 01:00:04.905426 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:04.905455 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:42945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 01:00:04.905656 10244 catalog_manager.cc:5495] T 6cd604fa75514a0a8fa8561f52e4da41 P e6356827c4004b85bf0c0009f1e84df5 reported cstate change: term changed from 0 to 1, leader changed from <none> to e6356827c4004b85bf0c0009f1e84df5 (127.3.75.2). New cstate: current_term: 1 leader_uuid: "e6356827c4004b85bf0c0009f1e84df5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a6cffb4a6a4ab3bc6f18c12a63b38e" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 44197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6356827c4004b85bf0c0009f1e84df5" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 44115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c7ac97c190134edfbd953cc9e2501843" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 33259 } health_report { overall_health: UNKNOWN } } }
I20260926 01:00:04.914566 10786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:04.912613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35318 (local address 127.3.75.2:44115)
0926 01:00:04.912668 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:04.912670 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:04.912693 (+ 23us) server_negotiation.cc:407] Connection header received
0926 01:00:04.912728 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:04.912728 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:04.912745 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:04.912799 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:00:04.912908 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:04.913339 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:04.913990 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:04.914154 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:04.914199 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:04.914350 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:00:04.914441 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:00:04.914492 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:04.914504 (+ 12us) server_negotiation.cc:299] Negotiation successful
0926 01:00:04.914520 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 01:00:04.915557 10652 raft_consensus.cc:1261] T 6cd604fa75514a0a8fa8561f52e4da41 P 17a6cffb4a6a4ab3bc6f18c12a63b38e [term 1 FOLLOWER]: Refusing update from remote peer e6356827c4004b85bf0c0009f1e84df5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:00:04.915694 10380 raft_consensus.cc:1261] T 6cd604fa75514a0a8fa8561f52e4da41 P c7ac97c190134edfbd953cc9e2501843 [term 1 FOLLOWER]: Refusing update from remote peer e6356827c4004b85bf0c0009f1e84df5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:00:04.915987 10801 consensus_queue.cc:1035] T 6cd604fa75514a0a8fa8561f52e4da41 P e6356827c4004b85bf0c0009f1e84df5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7ac97c190134edfbd953cc9e2501843" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 33259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:00:04.916144 10798 consensus_queue.cc:1035] T 6cd604fa75514a0a8fa8561f52e4da41 P e6356827c4004b85bf0c0009f1e84df5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17a6cffb4a6a4ab3bc6f18c12a63b38e" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 44197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:00:04.916626 10699 heartbeater.cc:503] Master 127.3.75.60:38525 was elected leader, sending a full tablet report...
I20260926 01:00:04.917078 10796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:04.916942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:42945 (local address 127.3.75.3:55479)
0926 01:00:04.916990 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:04.917039 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:42945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260926 01:00:04.917551 10797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:04.916941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:42945 (local address 127.3.75.1:34053)
0926 01:00:04.917491 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:04.917521 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:42945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":521}
I20260926 01:00:04.918274 10796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:04.918151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:42945 (local address 127.3.75.3:55035)
0926 01:00:04.918209 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:04.918235 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:42945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 01:00:04.918450 10800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:04.918242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:42945 (local address 127.3.75.2:34405)
0926 01:00:04.918396 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:04.918426 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:42945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260926 01:00:04.918919 10797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:04.918781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:42945 (local address 127.3.75.1:54091)
0926 01:00:04.918848 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:04.918892 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:42945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 01:00:04.919109 10807 mvcc.cc:205] Tried to move back new op lower bound from 7333414522532442112 to 7333414522491097088. Current Snapshot: MvccSnapshot[applied={T|T < 7333414522532442112}]
I20260926 01:00:04.920398 10244 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:00:04.920487 10244 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:00:04.921707 10244 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a945a1b3df7b4b8684f9b198beca1c89: Sending DeleteTablet for 3 replicas of tablet 6cd604fa75514a0a8fa8561f52e4da41
I20260926 01:00:04.922014 10360 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cd604fa75514a0a8fa8561f52e4da41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:00:04 UTC) from {username='oryx', principal='oryx/127.3.75.60@KRBTEST.COM'} at 127.0.0.1:50070
I20260926 01:00:04.922080 10496 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cd604fa75514a0a8fa8561f52e4da41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:00:04 UTC) from {username='oryx', principal='oryx/127.3.75.60@KRBTEST.COM'} at 127.0.0.1:35316
I20260926 01:00:04.922187 10814 tablet_replica.cc:330] T 6cd604fa75514a0a8fa8561f52e4da41 P c7ac97c190134edfbd953cc9e2501843: stopping tablet replica
I20260926 01:00:04.922179 10632 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cd604fa75514a0a8fa8561f52e4da41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:00:04 UTC) from {username='oryx', principal='oryx/127.3.75.60@KRBTEST.COM'} at 127.0.0.1:41676
I20260926 01:00:04.922235 10814 raft_consensus.cc:2229] T 6cd604fa75514a0a8fa8561f52e4da41 P c7ac97c190134edfbd953cc9e2501843 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:00:04.922297 10814 raft_consensus.cc:2258] T 6cd604fa75514a0a8fa8561f52e4da41 P c7ac97c190134edfbd953cc9e2501843 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:00:04.922386 10816 tablet_replica.cc:330] T 6cd604fa75514a0a8fa8561f52e4da41 P 17a6cffb4a6a4ab3bc6f18c12a63b38e: stopping tablet replica
I20260926 01:00:04.922442 10816 raft_consensus.cc:2229] T 6cd604fa75514a0a8fa8561f52e4da41 P 17a6cffb4a6a4ab3bc6f18c12a63b38e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:00:04.922487 10816 raft_consensus.cc:2258] T 6cd604fa75514a0a8fa8561f52e4da41 P 17a6cffb4a6a4ab3bc6f18c12a63b38e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:00:04.922586 10814 ts_tablet_manager.cc:1905] T 6cd604fa75514a0a8fa8561f52e4da41 P c7ac97c190134edfbd953cc9e2501843: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:00:04.922731 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 10297
I20260926 01:00:04.922830 10816 ts_tablet_manager.cc:1905] T 6cd604fa75514a0a8fa8561f52e4da41 P 17a6cffb4a6a4ab3bc6f18c12a63b38e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:00:04.924499 10816 ts_tablet_manager.cc:1918] T 6cd604fa75514a0a8fa8561f52e4da41 P 17a6cffb4a6a4ab3bc6f18c12a63b38e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:00:04.924552 10816 log.cc:1196] T 6cd604fa75514a0a8fa8561f52e4da41 P 17a6cffb4a6a4ab3bc6f18c12a63b38e: Deleting WAL directory at /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384372073861-3372-0/minicluster-data/ts-2/wal/wals/6cd604fa75514a0a8fa8561f52e4da41
I20260926 01:00:04.924782 10816 ts_tablet_manager.cc:1939] T 6cd604fa75514a0a8fa8561f52e4da41 P 17a6cffb4a6a4ab3bc6f18c12a63b38e: Deleting consensus metadata
I20260926 01:00:04.925132 10229 catalog_manager.cc:4841] TS 17a6cffb4a6a4ab3bc6f18c12a63b38e (127.3.75.3:44197): tablet 6cd604fa75514a0a8fa8561f52e4da41 (table test-table [id=629726b115724d02a67916e548714753]) successfully deleted
I20260926 01:00:04.930428 10815 tablet_replica.cc:330] T 6cd604fa75514a0a8fa8561f52e4da41 P e6356827c4004b85bf0c0009f1e84df5: stopping tablet replica
I20260926 01:00:04.930508 10815 raft_consensus.cc:2229] T 6cd604fa75514a0a8fa8561f52e4da41 P e6356827c4004b85bf0c0009f1e84df5 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:00:04.930606 10815 raft_consensus.cc:2258] T 6cd604fa75514a0a8fa8561f52e4da41 P e6356827c4004b85bf0c0009f1e84df5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:00:04.930747 10785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:04.930512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.1:33259 (local address 127.0.0.1:50084)
0926 01:00:04.930666 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:04.930695 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.1:33259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
W20260926 01:00:04.930922 10232 catalog_manager.cc:4584] TS c7ac97c190134edfbd953cc9e2501843 (127.3.75.1:33259): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6cd604fa75514a0a8fa8561f52e4da41: Network error: Client connection negotiation failed: client connection to 127.3.75.1:33259: connect: Connection refused (error 111)
I20260926 01:00:04.930977 10815 ts_tablet_manager.cc:1905] T 6cd604fa75514a0a8fa8561f52e4da41 P e6356827c4004b85bf0c0009f1e84df5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:00:04.931090 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 10433
I20260926 01:00:04.936177 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 10569
I20260926 01:00:04.936210 10785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:04.936095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.2:44115 (local address 127.0.0.1:35330)
0926 01:00:04.936151 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:04.936183 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.2:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 01:00:04.940914 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 9926
I20260926 01:00:04.945588 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 10113
2026-09-26T01:00:04Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4617 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:00:05 dist-test-slave-7v10 krb5kdc[10819](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:00:05 dist-test-slave-7v10 krb5kdc[10819](info): set up 2 sockets
Sep 26 01:00:05 dist-test-slave-7v10 krb5kdc[10819](info): commencing operation
krb5kdc: starting...
W20260926 01:00:06.034444 3372 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Sep 26 01:00:06 dist-test-slave-7v10 krb5kdc[10819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:00:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:00:06Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.75.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.75.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.75.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/krb5kdc/oryx_127.3.75.62.keytab.
Entry for principal oryx/127.3.75.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/krb5kdc/oryx_127.3.75.62.keytab.
I20260926 01:00:06.149403 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:33509
--webserver_interface=127.3.75.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/krb5kdc/oryx_127.3.75.62.keytab
--principal=oryx/127.3.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:39055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.75.62:33509
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:00:06.230355 10832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:00:06.230572 10832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:00:06.230595 10832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:00:06.231788 10832 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:00:06.231834 10832 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 01:00:06.231850 10832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:00:06.231914 10832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:00:06.231945 10832 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:00:06.231973 10832 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:00:06.233515 10832 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:39055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.75.62:33509
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/krb5kdc/oryx_127.3.75.62.keytab
--principal=oryx/127.3.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:33509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:00:06.233780 10832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:00:06.234036 10832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:00:06.238862 10840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:00:06.238867 10832 server_base.cc:1034] running on GCE node
W20260926 01:00:06.238839 10838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:00:06.238835 10837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:00:06.239416 10832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:00:06.239717 10832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:00:06.240901 10832 hybrid_clock.cc:648] HybridClock initialized: now 1790384406240883 us; error 37 us; skew 500 ppm
Sep 26 01:00:06 dist-test-slave-7v10 krb5kdc[10819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384406, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:00:06.243314 10832 init.cc:377] Logged in from keytab as oryx/127.3.75.62@KRBTEST.COM (short username oryx)
I20260926 01:00:06.243709 10832 webserver.cc:460] Webserver started at http://127.3.75.62:41621/ using document root <none> and password file <none>
I20260926 01:00:06.243883 10832 fs_manager.cc:359] Metadata directory not provided
I20260926 01:00:06.243921 10832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:00:06.244006 10832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:00:06.244951 10832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/master-0/data/instance:
uuid: "246badea3f0f4badadc5380fb63e9019"
format_stamp: "Formatted at 2026-09-26 01:00:06 on dist-test-slave-7v10"
server_key: "1638c0c76c6b2276b7a5c59b22081ef7"
server_key_iv: "5f8fb6e4dcecc28eb26206f982344dd5"
server_key_version: "encryptionkey@0"
I20260926 01:00:06.245255 10832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/master-0/wal/instance:
uuid: "246badea3f0f4badadc5380fb63e9019"
format_stamp: "Formatted at 2026-09-26 01:00:06 on dist-test-slave-7v10"
server_key: "1638c0c76c6b2276b7a5c59b22081ef7"
server_key_iv: "5f8fb6e4dcecc28eb26206f982344dd5"
server_key_version: "encryptionkey@0"
I20260926 01:00:06.246444 10832 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 01:00:06.247155 10847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:06.247305 10832 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:06.247393 10832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/master-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/master-0/wal
uuid: "246badea3f0f4badadc5380fb63e9019"
format_stamp: "Formatted at 2026-09-26 01:00:06 on dist-test-slave-7v10"
server_key: "1638c0c76c6b2276b7a5c59b22081ef7"
server_key_iv: "5f8fb6e4dcecc28eb26206f982344dd5"
server_key_version: "encryptionkey@0"
I20260926 01:00:06.247444 10832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:00:06.256749 10832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:00:06.258384 10832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:00:06.258486 10832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:00:06.262692 10832 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.62:33509
I20260926 01:00:06.262769 10909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.62:33509 every 8 connection(s)
I20260926 01:00:06.263141 10832 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
I20260926 01:00:06.263583 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 10832
I20260926 01:00:06.263680 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/master-0/wal/instance
I20260926 01:00:06.263906 3372 external_mini_cluster.cc:1428] Setting key 3c12eaed4641085c9d8fefb1082234dd
I20260926 01:00:06.263923 10910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:00:06.267184 10910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 246badea3f0f4badadc5380fb63e9019: Bootstrap starting.
I20260926 01:00:06.267840 10910 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 246badea3f0f4badadc5380fb63e9019: Neither blocks nor log segments found. Creating new log.
I20260926 01:00:06.268155 10910 log.cc:824] T 00000000000000000000000000000000 P 246badea3f0f4badadc5380fb63e9019: Log is configured to *not* fsync() on all Append() calls
I20260926 01:00:06.268851 10910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 246badea3f0f4badadc5380fb63e9019: No bootstrap required, opened a new log
Sep 26 01:00:06 dist-test-slave-7v10 krb5kdc[10819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM
I20260926 01:00:06.271037 10910 raft_consensus.cc:348] T 00000000000000000000000000000000 P 246badea3f0f4badadc5380fb63e9019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "246badea3f0f4badadc5380fb63e9019" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 33509 } }
I20260926 01:00:06.271165 10910 raft_consensus.cc:374] T 00000000000000000000000000000000 P 246badea3f0f4badadc5380fb63e9019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:00:06.271188 10910 raft_consensus.cc:729] T 00000000000000000000000000000000 P 246badea3f0f4badadc5380fb63e9019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 246badea3f0f4badadc5380fb63e9019, State: Initialized, Role: FOLLOWER
I20260926 01:00:06.271325 10910 consensus_queue.cc:260] T 00000000000000000000000000000000 P 246badea3f0f4badadc5380fb63e9019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "246badea3f0f4badadc5380fb63e9019" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 33509 } }
I20260926 01:00:06.271399 10910 raft_consensus.cc:388] T 00000000000000000000000000000000 P 246badea3f0f4badadc5380fb63e9019 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:00:06.271421 10910 raft_consensus.cc:482] T 00000000000000000000000000000000 P 246badea3f0f4badadc5380fb63e9019 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:00:06.271484 10910 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 246badea3f0f4badadc5380fb63e9019 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:00:06.271988 10910 raft_consensus.cc:504] T 00000000000000000000000000000000 P 246badea3f0f4badadc5380fb63e9019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "246badea3f0f4badadc5380fb63e9019" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 33509 } }
I20260926 01:00:06.272074 10910 leader_election.cc:302] T 00000000000000000000000000000000 P 246badea3f0f4badadc5380fb63e9019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 246badea3f0f4badadc5380fb63e9019; no voters:
I20260926 01:00:06.272306 10910 leader_election.cc:288] T 00000000000000000000000000000000 P 246badea3f0f4badadc5380fb63e9019 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:00:06.272333 10915 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 246badea3f0f4badadc5380fb63e9019 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:00:06.272508 10915 raft_consensus.cc:686] T 00000000000000000000000000000000 P 246badea3f0f4badadc5380fb63e9019 [term 1 LEADER]: Becoming Leader. State: Replica: 246badea3f0f4badadc5380fb63e9019, State: Running, Role: LEADER
I20260926 01:00:06.272639 10915 consensus_queue.cc:237] T 00000000000000000000000000000000 P 246badea3f0f4badadc5380fb63e9019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "246badea3f0f4badadc5380fb63e9019" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 33509 } }
I20260926 01:00:06.272637 10910 sys_catalog.cc:564] T 00000000000000000000000000000000 P 246badea3f0f4badadc5380fb63e9019 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:00:06.273082 10916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 246badea3f0f4badadc5380fb63e9019 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "246badea3f0f4badadc5380fb63e9019" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "246badea3f0f4badadc5380fb63e9019" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 33509 } } }
I20260926 01:00:06.273345 10916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 246badea3f0f4badadc5380fb63e9019 [sys.catalog]: This master's current role is: LEADER
I20260926 01:00:06.273264 10917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 246badea3f0f4badadc5380fb63e9019 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 246badea3f0f4badadc5380fb63e9019. Latest consensus state: current_term: 1 leader_uuid: "246badea3f0f4badadc5380fb63e9019" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "246badea3f0f4badadc5380fb63e9019" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 33509 } } }
I20260926 01:00:06.273814 10917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 246badea3f0f4badadc5380fb63e9019 [sys.catalog]: This master's current role is: LEADER
I20260926 01:00:06.273741 10920 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:00:06.274536 10920 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:00:06.275110 10913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:06.264430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59024 (local address 127.3.75.62:33509)
0926 01:00:06.264616 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:06.264626 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:06.264656 (+ 30us) server_negotiation.cc:407] Connection header received
0926 01:00:06.265071 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:06.265080 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:06.265138 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:06.265295 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:06.265886 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:06.267334 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:06.268100 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:06.268647 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:06.268748 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:06.271315 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:06.271320 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:06.271320 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:00:06.271334 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:06.273380 (+ 2046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:06.273831 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:06.273834 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:06.273835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:06.273939 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:06.274161 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:06.274162 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:06.274162 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:06.274475 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:06.274635 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:06.274744 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 01:00:06.274911 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":64,"threads_started":1}
I20260926 01:00:06.276085 10920 catalog_manager.cc:1347] Generated new cluster ID: bc886333e351441088540290d8a018f6
I20260926 01:00:06.276134 10920 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:00:06.283277 10920 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:00:06.283968 10920 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:00:06.288000 10920 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 246badea3f0f4badadc5380fb63e9019: Generated new TSK 0
I20260926 01:00:06.288180 10920 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.75.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.75.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.75.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/krb5kdc/oryx_127.3.75.1.keytab.
Entry for principal oryx/127.3.75.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/krb5kdc/oryx_127.3.75.1.keytab.
I20260926 01:00:06.315318 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.1:0
--local_ip_for_outbound_sockets=127.3.75.1
--webserver_interface=127.3.75.1
--webserver_port=0
--tserver_master_addrs=127.3.75.62:33509
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/krb5kdc/oryx_127.3.75.1.keytab
--principal=oryx/127.3.75.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:39055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:00:06.394718 10936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:00:06.395026 10936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:00:06.395073 10936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:00:06.396261 10936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:00:06.396329 10936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:00:06.396363 10936 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 01:00:06.396400 10936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.1
I20260926 01:00:06.397929 10936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:39055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/krb5kdc/oryx_127.3.75.1.keytab
--principal=oryx/127.3.75.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.75.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.75.62:33509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.1
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:00:06.398203 10936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:00:06.398451 10936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:00:06.403667 10942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:00:06.403690 10941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:00:06.403693 10944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:00:06.403936 10936 server_base.cc:1034] running on GCE node
I20260926 01:00:06.404184 10936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:00:06.404385 10936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:00:06.405524 10936 hybrid_clock.cc:648] HybridClock initialized: now 1790384406405508 us; error 31 us; skew 500 ppm
Sep 26 01:00:06 dist-test-slave-7v10 krb5kdc[10819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384406, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.75.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:00:06.407990 10936 init.cc:377] Logged in from keytab as oryx/127.3.75.1@KRBTEST.COM (short username oryx)
I20260926 01:00:06.408375 10936 webserver.cc:460] Webserver started at http://127.3.75.1:37291/ using document root <none> and password file <none>
I20260926 01:00:06.408568 10936 fs_manager.cc:359] Metadata directory not provided
I20260926 01:00:06.408608 10936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:00:06.408764 10936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:00:06.409865 10936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/ts-0/data/instance:
uuid: "78b75a2199c0452f9bbac4eb3746b057"
format_stamp: "Formatted at 2026-09-26 01:00:06 on dist-test-slave-7v10"
server_key: "1db729057ac008261e1cd85706088c5c"
server_key_iv: "afee7bc3ca24f53aa7d07dc5868ba46a"
server_key_version: "encryptionkey@0"
I20260926 01:00:06.410212 10936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/ts-0/wal/instance:
uuid: "78b75a2199c0452f9bbac4eb3746b057"
format_stamp: "Formatted at 2026-09-26 01:00:06 on dist-test-slave-7v10"
server_key: "1db729057ac008261e1cd85706088c5c"
server_key_iv: "afee7bc3ca24f53aa7d07dc5868ba46a"
server_key_version: "encryptionkey@0"
I20260926 01:00:06.411573 10936 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 01:00:06.412572 10951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:06.412766 10936 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:00:06.412860 10936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/ts-0/wal
uuid: "78b75a2199c0452f9bbac4eb3746b057"
format_stamp: "Formatted at 2026-09-26 01:00:06 on dist-test-slave-7v10"
server_key: "1db729057ac008261e1cd85706088c5c"
server_key_iv: "afee7bc3ca24f53aa7d07dc5868ba46a"
server_key_version: "encryptionkey@0"
I20260926 01:00:06.412936 10936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:00:06.451035 10936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:00:06.452970 10936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:00:06.453150 10936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:00:06.453688 10936 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:00:06.453735 10936 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:06.453769 10936 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:00:06.453821 10936 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 01:00:06 dist-test-slave-7v10 krb5kdc[10819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384406, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.75.1@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM
I20260926 01:00:06.461004 10936 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.1:45301
I20260926 01:00:06.461071 11066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.1:45301 every 8 connection(s)
I20260926 01:00:06.461413 10936 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
I20260926 01:00:06.464179 10913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:06.454417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:35263 (local address 127.3.75.62:33509)
0926 01:00:06.454509 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:06.454511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:06.454703 (+ 192us) server_negotiation.cc:407] Connection header received
0926 01:00:06.455293 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:06.455294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:06.455309 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:06.455383 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:06.456264 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:06.457069 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:06.458576 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:06.458962 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:06.459052 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:06.461452 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:06.461454 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:06.461454 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:00:06.461463 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:06.462997 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:06.463397 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:06.463398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:06.463399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:06.463456 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:06.463684 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:06.463685 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:06.463685 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:06.463794 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:06.463863 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:06.464055 (+ 192us) server_negotiation.cc:299] Negotiation successful
0926 01:00:06.464118 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 01:00:06.464485 10960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:06.454383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:33509 (local address 127.3.75.1:35263)
0926 01:00:06.454570 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:06.454610 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:06.455170 (+ 560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:06.455517 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:06.455522 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:06.455882 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:06.456151 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:06.456156 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:06.457172 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:06.457173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:06.458479 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:06.458481 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:06.459049 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:06.459050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:06.459152 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:06.459630 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:06.459638 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:06.461352 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:00:06.463080 (+ 1728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:06.463086 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:06.463088 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:06.463303 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:06.463524 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:06.463524 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:06.463525 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:06.463628 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:06.463853 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:00:06.463855 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:00:06.463973 (+ 118us) client_negotiation.cc:769] Sending connection context
0926 01:00:06.464043 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 01:00:06.464148 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":117,"threads_started":1}
I20260926 01:00:06.465032 11067 heartbeater.cc:347] Connected to a master server at 127.3.75.62:33509
I20260926 01:00:06.465116 11067 heartbeater.cc:464] Registering TS with master...
I20260926 01:00:06.465301 11067 heartbeater.cc:511] Master 127.3.75.62:33509 requested a full tablet report, sending...
I20260926 01:00:06.465842 10863 ts_manager.cc:194] Registered new tserver with Master: 78b75a2199c0452f9bbac4eb3746b057 (127.3.75.1:45301)
I20260926 01:00:06.466749 10863 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.75.1@KRBTEST.COM'} at 127.3.75.1:35263
I20260926 01:00:06.470185 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 10936
I20260926 01:00:06.470299 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/ts-0/wal/instance
I20260926 01:00:06.470502 3372 external_mini_cluster.cc:1428] Setting key 379d032f50ea220c3436f27d2c22a676
I20260926 01:00:06.471267 10913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:06.466019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:44213 (local address 127.3.75.62:33509)
0926 01:00:06.466072 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:06.466074 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:06.466205 (+ 131us) server_negotiation.cc:407] Connection header received
0926 01:00:06.466259 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:06.466260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:06.466269 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:06.466323 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:06.466573 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:06.467239 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:06.467898 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:06.468082 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:06.468134 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:06.468958 (+ 824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:06.468959 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:06.468959 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:00:06.468963 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:06.470039 (+ 1076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:06.470372 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:06.470373 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:06.470373 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:06.470434 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:06.470780 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:06.470781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:06.470781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:06.470939 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:06.471008 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:06.471107 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 01:00:06.471174 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 01:00:06.471277 11073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:06.466023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:33509 (local address 127.3.75.1:44213)
0926 01:00:06.466139 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:06.466148 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:06.466203 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:06.466326 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:06.466326 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:06.466475 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:06.466521 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:06.466522 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:06.467334 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:06.467335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:06.467808 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:06.467809 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:06.468144 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:06.468145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:06.468209 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:06.468626 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:06.468635 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:06.468893 (+ 258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:00:06.470118 (+ 1225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:06.470119 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:06.470119 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:06.470292 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:06.470509 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:06.470509 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:06.470510 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:06.470690 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:06.471000 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:00:06.471001 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:00:06.471034 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 01:00:06.471080 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 01:00:06.471146 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":71,"threads_started":1}
WARNING: no policy specified for oryx/127.3.75.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.75.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.75.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/krb5kdc/oryx_127.3.75.2.keytab.
Entry for principal oryx/127.3.75.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/krb5kdc/oryx_127.3.75.2.keytab.
I20260926 01:00:06.493758 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.2:0
--local_ip_for_outbound_sockets=127.3.75.2
--webserver_interface=127.3.75.2
--webserver_port=0
--tserver_master_addrs=127.3.75.62:33509
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/krb5kdc/oryx_127.3.75.2.keytab
--principal=oryx/127.3.75.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:39055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:00:06.578941 11076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:00:06.579219 11076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:00:06.579246 11076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:00:06.580523 11076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:00:06.580575 11076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:00:06.580657 11076 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 01:00:06.580710 11076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.2
I20260926 01:00:06.582494 11076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:39055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/krb5kdc/oryx_127.3.75.2.keytab
--principal=oryx/127.3.75.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.75.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.75.62:33509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.2
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:00:06.582788 11076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:00:06.583097 11076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:00:06.588809 11081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:00:06.588838 11082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:00:06.588943 11076 server_base.cc:1034] running on GCE node
W20260926 01:00:06.588814 11084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:00:06.589257 11076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:00:06.589448 11076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:00:06.590631 11076 hybrid_clock.cc:648] HybridClock initialized: now 1790384406590621 us; error 33 us; skew 500 ppm
Sep 26 01:00:06 dist-test-slave-7v10 krb5kdc[10819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384406, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.75.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:00:06.592978 11076 init.cc:377] Logged in from keytab as oryx/127.3.75.2@KRBTEST.COM (short username oryx)
I20260926 01:00:06.593434 11076 webserver.cc:460] Webserver started at http://127.3.75.2:42921/ using document root <none> and password file <none>
I20260926 01:00:06.593667 11076 fs_manager.cc:359] Metadata directory not provided
I20260926 01:00:06.593745 11076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:00:06.593894 11076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:00:06.594976 11076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/ts-1/data/instance:
uuid: "a42f67fb6fef43c6867b287287170a1f"
format_stamp: "Formatted at 2026-09-26 01:00:06 on dist-test-slave-7v10"
server_key: "b0375ef957c7295219957d1b58778813"
server_key_iv: "93fc946d5dd3c84d5d6a6bfb35247042"
server_key_version: "encryptionkey@0"
I20260926 01:00:06.595333 11076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/ts-1/wal/instance:
uuid: "a42f67fb6fef43c6867b287287170a1f"
format_stamp: "Formatted at 2026-09-26 01:00:06 on dist-test-slave-7v10"
server_key: "b0375ef957c7295219957d1b58778813"
server_key_iv: "93fc946d5dd3c84d5d6a6bfb35247042"
server_key_version: "encryptionkey@0"
I20260926 01:00:06.596601 11076 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:00:06.597324 11091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:06.597489 11076 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:06.597589 11076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/ts-1/wal
uuid: "a42f67fb6fef43c6867b287287170a1f"
format_stamp: "Formatted at 2026-09-26 01:00:06 on dist-test-slave-7v10"
server_key: "b0375ef957c7295219957d1b58778813"
server_key_iv: "93fc946d5dd3c84d5d6a6bfb35247042"
server_key_version: "encryptionkey@0"
I20260926 01:00:06.597671 11076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:00:06.619290 11076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:00:06.621135 11076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:00:06.621299 11076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:00:06.621865 11076 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:00:06.621927 11076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:06.621984 11076 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:00:06.622032 11076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 01:00:06 dist-test-slave-7v10 krb5kdc[10819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384406, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.75.2@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM
I20260926 01:00:06.629381 11076 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.2:39043
I20260926 01:00:06.629523 11206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.2:39043 every 8 connection(s)
I20260926 01:00:06.629774 11076 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
I20260926 01:00:06.631302 10913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:06.622277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:40253 (local address 127.3.75.62:33509)
0926 01:00:06.622349 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:06.622351 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:06.622581 (+ 230us) server_negotiation.cc:407] Connection header received
0926 01:00:06.623115 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:06.623116 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:06.623131 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:06.623209 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:06.623927 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:06.624502 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:06.625526 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:06.625735 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:06.625834 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:06.628686 (+ 2852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:06.628687 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:06.628688 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:00:06.628695 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:06.630113 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:06.630508 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:06.630509 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:06.630509 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:06.630566 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:06.630740 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:06.630740 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:06.630741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:06.630833 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:06.630930 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:06.631157 (+ 227us) server_negotiation.cc:299] Negotiation successful
0926 01:00:06.631239 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 01:00:06.631443 11100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:06.622278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:33509 (local address 127.3.75.2:40253)
0926 01:00:06.622443 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:06.622482 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:06.623011 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:06.623288 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:06.623291 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:06.623626 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:06.623820 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:06.623825 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:06.624618 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:06.624619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:06.625429 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:06.625431 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:06.625844 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:06.625845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:06.625983 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:06.626519 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:06.626527 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:06.628567 (+ 2040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:00:06.630206 (+ 1639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:06.630211 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:06.630213 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:06.630432 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:06.630614 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:06.630615 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:06.630615 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:06.630702 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:06.630925 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:00:06.630929 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:00:06.631042 (+ 113us) client_negotiation.cc:769] Sending connection context
0926 01:00:06.631123 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 01:00:06.631236 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":99,"threads_started":1}
I20260926 01:00:06.631953 11207 heartbeater.cc:347] Connected to a master server at 127.3.75.62:33509
I20260926 01:00:06.632033 11207 heartbeater.cc:464] Registering TS with master...
I20260926 01:00:06.632220 11207 heartbeater.cc:511] Master 127.3.75.62:33509 requested a full tablet report, sending...
I20260926 01:00:06.632642 10863 ts_manager.cc:194] Registered new tserver with Master: a42f67fb6fef43c6867b287287170a1f (127.3.75.2:39043)
I20260926 01:00:06.633105 10863 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.75.2@KRBTEST.COM'} at 127.3.75.2:40253
I20260926 01:00:06.638212 10913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:06.633092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:39607 (local address 127.3.75.62:33509)
0926 01:00:06.633128 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:06.633130 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:06.633210 (+ 80us) server_negotiation.cc:407] Connection header received
0926 01:00:06.633311 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:06.633312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:06.633321 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:06.633370 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:06.633697 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:06.634149 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:06.634833 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:06.635069 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:06.635155 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:06.635982 (+ 827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:06.635984 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:06.635984 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:00:06.635990 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:06.637074 (+ 1084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:06.637418 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:06.637419 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:06.637419 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:06.637478 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:06.637673 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:06.637674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:06.637674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:06.637824 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:06.637908 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:06.638054 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 01:00:06.638127 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 01:00:06.638224 11213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:06.632997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:33509 (local address 127.3.75.2:39607)
0926 01:00:06.633141 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:06.633153 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:06.633235 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:06.633372 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:06.633373 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:06.633559 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:06.633622 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:06.633623 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:06.634250 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:06.634250 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:06.634738 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:06.634739 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:06.635158 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:06.635159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:06.635226 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:06.635580 (+ 354us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:06.635585 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:06.635893 (+ 308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:00:06.637172 (+ 1279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:06.637173 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:06.637173 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:06.637343 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:06.637549 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:06.637550 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:06.637550 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:06.637607 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:06.637900 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:00:06.637901 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:00:06.637943 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 01:00:06.638029 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 01:00:06.638124 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":90,"threads_started":1}
I20260926 01:00:06.638476 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 11076
I20260926 01:00:06.638583 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/ts-1/wal/instance
I20260926 01:00:06.638773 3372 external_mini_cluster.cc:1428] Setting key 9a1d74d37ded037833bf5731725da239
WARNING: no policy specified for oryx/127.3.75.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.75.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.75.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/krb5kdc/oryx_127.3.75.3.keytab.
Entry for principal oryx/127.3.75.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/krb5kdc/oryx_127.3.75.3.keytab.
I20260926 01:00:06.665688 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.3:0
--local_ip_for_outbound_sockets=127.3.75.3
--webserver_interface=127.3.75.3
--webserver_port=0
--tserver_master_addrs=127.3.75.62:33509
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/krb5kdc/oryx_127.3.75.3.keytab
--principal=oryx/127.3.75.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:39055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:00:06.746477 11216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:00:06.746667 11216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:00:06.746688 11216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:00:06.747941 11216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:00:06.747982 11216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:00:06.748003 11216 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 01:00:06.748019 11216 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.3
I20260926 01:00:06.749500 11216 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:39055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/krb5kdc/oryx_127.3.75.3.keytab
--principal=oryx/127.3.75.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.75.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.75.62:33509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11216
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.3
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:00:06.749794 11216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:00:06.750041 11216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:00:06.755129 11221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:00:06.755138 11222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:00:06.755367 11216 server_base.cc:1034] running on GCE node
W20260926 01:00:06.755218 11224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:00:06.755666 11216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:00:06.755860 11216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:00:06.757045 11216 hybrid_clock.cc:648] HybridClock initialized: now 1790384406757031 us; error 44 us; skew 500 ppm
Sep 26 01:00:06 dist-test-slave-7v10 krb5kdc[10819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384406, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.75.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:00:06.759521 11216 init.cc:377] Logged in from keytab as oryx/127.3.75.3@KRBTEST.COM (short username oryx)
I20260926 01:00:06.759966 11216 webserver.cc:460] Webserver started at http://127.3.75.3:39261/ using document root <none> and password file <none>
I20260926 01:00:06.760166 11216 fs_manager.cc:359] Metadata directory not provided
I20260926 01:00:06.760208 11216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:00:06.760300 11216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:00:06.761273 11216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/ts-2/data/instance:
uuid: "47a2277a9fd24c14b23f901d6823409f"
format_stamp: "Formatted at 2026-09-26 01:00:06 on dist-test-slave-7v10"
server_key: "32929cd91bf2a490dd3c16c90a43291b"
server_key_iv: "d2bfa83889227f360c7a0e4f06b015da"
server_key_version: "encryptionkey@0"
I20260926 01:00:06.761574 11216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/ts-2/wal/instance:
uuid: "47a2277a9fd24c14b23f901d6823409f"
format_stamp: "Formatted at 2026-09-26 01:00:06 on dist-test-slave-7v10"
server_key: "32929cd91bf2a490dd3c16c90a43291b"
server_key_iv: "d2bfa83889227f360c7a0e4f06b015da"
server_key_version: "encryptionkey@0"
I20260926 01:00:06.762821 11216 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:00:06.763566 11231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:06.763742 11216 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 01:00:06.763837 11216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/ts-2/wal
uuid: "47a2277a9fd24c14b23f901d6823409f"
format_stamp: "Formatted at 2026-09-26 01:00:06 on dist-test-slave-7v10"
server_key: "32929cd91bf2a490dd3c16c90a43291b"
server_key_iv: "d2bfa83889227f360c7a0e4f06b015da"
server_key_version: "encryptionkey@0"
I20260926 01:00:06.763924 11216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:00:06.785635 11216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:00:06.787621 11216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:00:06.787791 11216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:00:06.788645 11216 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:00:06.788698 11216 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:06.788734 11216 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:00:06.788756 11216 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 01:00:06 dist-test-slave-7v10 krb5kdc[10819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384406, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.75.3@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM
I20260926 01:00:06.796042 11216 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.3:46757
I20260926 01:00:06.796393 11346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.3:46757 every 8 connection(s)
I20260926 01:00:06.796401 11216 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
I20260926 01:00:06.798357 10913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:06.788865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.3:49647 (local address 127.3.75.62:33509)
0926 01:00:06.788926 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:06.788929 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:06.789147 (+ 218us) server_negotiation.cc:407] Connection header received
0926 01:00:06.789649 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:06.789651 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:06.789684 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:06.789775 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:06.790403 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:06.791052 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:06.792132 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:06.792370 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:06.792468 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:06.795483 (+ 3015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:06.795484 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:06.795484 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:00:06.795492 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:06.797149 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:06.797554 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:06.797554 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:06.797555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:06.797606 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:06.797823 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:06.797823 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:06.797824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:06.797921 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:06.797970 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:06.798235 (+ 265us) server_negotiation.cc:299] Negotiation successful
0926 01:00:06.798295 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 01:00:06.798570 11240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:06.788849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:33509 (local address 127.3.75.3:49647)
0926 01:00:06.789002 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:06.789048 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:06.789550 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:06.789836 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:06.789839 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:06.790063 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:06.790295 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:06.790300 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:06.791147 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:06.791148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:06.792021 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:06.792023 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:06.792475 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:06.792477 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:06.792615 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:06.793145 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:06.793154 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:06.795353 (+ 2199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:00:06.797246 (+ 1893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:06.797252 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:06.797253 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:06.797480 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:06.797668 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:06.797668 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:06.797669 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:06.797762 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:06.797978 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:00:06.797982 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:00:06.798086 (+ 104us) client_negotiation.cc:769] Sending connection context
0926 01:00:06.798203 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 01:00:06.798317 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":58,"threads_started":1}
I20260926 01:00:06.799081 11347 heartbeater.cc:347] Connected to a master server at 127.3.75.62:33509
I20260926 01:00:06.799167 11347 heartbeater.cc:464] Registering TS with master...
I20260926 01:00:06.799347 11347 heartbeater.cc:511] Master 127.3.75.62:33509 requested a full tablet report, sending...
I20260926 01:00:06.799806 10864 ts_manager.cc:194] Registered new tserver with Master: 47a2277a9fd24c14b23f901d6823409f (127.3.75.3:46757)
I20260926 01:00:06.800325 10864 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.75.3@KRBTEST.COM'} at 127.3.75.3:49647
I20260926 01:00:06.800527 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 11216
I20260926 01:00:06.800611 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/ts-2/wal/instance
I20260926 01:00:06.800801 3372 external_mini_cluster.cc:1428] Setting key 18b8b6f331d88ebaf7163ce320690331
I20260926 01:00:06.802124 3372 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 01:00:06.806180 11353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:06.800130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:33509 (local address 127.3.75.3:53163)
0926 01:00:06.800253 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:06.800263 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:06.800343 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:06.800562 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:06.800563 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:06.800797 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:06.800925 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:06.800927 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:06.801720 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:06.801720 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:06.802208 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:06.802210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:06.802605 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:06.802606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:06.802677 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:06.803160 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:06.803165 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:06.803432 (+ 267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:00:06.805174 (+ 1742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:06.805175 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:06.805175 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:06.805347 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:06.805554 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:06.805555 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:06.805555 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:06.805605 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:06.805877 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:00:06.805877 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:00:06.805908 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 01:00:06.805959 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 01:00:06.806028 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":66,"threads_started":1}
I20260926 01:00:06.806185 10913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:06.800182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.3:53163 (local address 127.3.75.62:33509)
0926 01:00:06.800218 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:06.800220 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:06.800333 (+ 113us) server_negotiation.cc:407] Connection header received
0926 01:00:06.800461 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:06.800463 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:06.800474 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:06.800504 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:06.801027 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:06.801635 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:06.802303 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:06.802513 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:06.802603 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:06.803541 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:06.803543 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:06.803543 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:00:06.803550 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:06.805083 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:06.805426 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:06.805427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:06.805427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:06.805486 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:06.805670 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:06.805671 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:06.805672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:06.805811 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:06.805880 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:06.805983 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 01:00:06.806063 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
Sep 26 01:00:06 dist-test-slave-7v10 krb5kdc[10819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384406, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 01:00:06 dist-test-slave-7v10 krb5kdc[10819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:00:06 dist-test-slave-7v10 krb5kdc[10819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 01:00:06.824154 10913 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:00:06.818905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59030 (local address 127.3.75.62:33509)
0926 01:00:06.818947 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:06.818949 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:06.818985 (+ 36us) server_negotiation.cc:407] Connection header received
0926 01:00:06.819048 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:06.819049 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:06.819056 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:06.819102 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:06.819645 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:06.820086 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:06.820797 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:06.820999 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:06.821046 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:06.824065 (+ 3019us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59030: BlockingRecv error: recv got EOF from 127.0.0.1:59030 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 26 01:00:06 dist-test-slave-7v10 krb5kdc[10819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384406, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM
I20260926 01:00:06.830996 10913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:06.824930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59038 (local address 127.3.75.62:33509)
0926 01:00:06.824986 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:06.824987 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:06.825061 (+ 74us) server_negotiation.cc:407] Connection header received
0926 01:00:06.825094 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:06.825094 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:06.825101 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:06.825143 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:06.825539 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:06.825956 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:06.826509 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:06.826655 (+ 146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:06.826731 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:06.828731 (+ 2000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:06.828732 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:06.828732 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:00:06.828737 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:06.830098 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:06.830378 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:06.830379 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:06.830379 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:06.830424 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:06.830564 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:06.830564 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:06.830564 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:06.830642 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:06.830703 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:06.830793 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 01:00:06.830842 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 01:00:06.832769 10864 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59038:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:00:06.834395 10864 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:00:06.841192 11369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:06.836851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.1:45301 (local address 127.0.0.1:50308)
0926 01:00:06.837158 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:06.837167 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:06.837208 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:06.837517 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:06.837520 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:06.837522 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:00:06.837620 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:06.837622 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:06.838969 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:06.838970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:06.839961 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:06.839964 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:06.840922 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:06.840923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:06.840988 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:06.840999 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 01:00:06.841056 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 01:00:06.841091 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":55,"thread_start_us":41,"threads_started":1}
I20260926 01:00:06.841266 11371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:06.836981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50308 (local address 127.3.75.1:45301)
0926 01:00:06.837352 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:06.837357 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:06.837367 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:00:06.837406 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:06.837408 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:06.837423 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:06.837502 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:00:06.837707 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:06.838867 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:06.840054 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:06.840804 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:06.840903 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:06.840981 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:06.841099 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 01:00:06.841150 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":108,"threads_started":1}
I20260926 01:00:06.841715 11372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:06.837050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.2:39043 (local address 127.0.0.1:57088)
0926 01:00:06.837205 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:06.837213 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:06.837248 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:06.837819 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:06.837820 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:06.837822 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:00:06.837888 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:06.837889 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:06.838735 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:06.838736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:06.840169 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:06.840172 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:06.841167 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:06.841168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:06.841542 (+ 374us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:06.841550 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 01:00:06.841610 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 01:00:06.841652 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":30,"thread_start_us":65,"threads_started":1}
I20260926 01:00:06.842029 11373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:06.837177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57088 (local address 127.3.75.2:39043)
0926 01:00:06.837635 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:06.837639 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:06.837647 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:00:06.837680 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:06.837682 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:06.837710 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:06.837819 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:00:06.837964 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:06.838641 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:06.840264 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:06.841075 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:06.841147 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:06.841223 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:06.841882 (+ 659us) server_negotiation.cc:299] Negotiation successful
0926 01:00:06.841930 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":100,"threads_started":1}
I20260926 01:00:06.841943 11001 tablet_service.cc:1467] Processing CreateTablet for tablet 92c2a7441b7947c191ff9b041f4181f0 (DEFAULT_TABLE table=test-table [id=6f90b9c491f447a7a113574d4551c7bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:00:06.842334 11001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92c2a7441b7947c191ff9b041f4181f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:00:06.842449 11370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:06.836851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.3:46757 (local address 127.0.0.1:42006)
0926 01:00:06.837001 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:06.837042 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:06.837103 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:06.837541 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:06.837542 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:06.837544 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:00:06.837611 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:06.837612 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:06.838982 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:06.838983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:06.840169 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:06.840172 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:06.842193 (+ 2021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:06.842194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:06.842260 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:06.842264 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:00:06.842333 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 01:00:06.842376 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":29,"thread_start_us":81,"threads_started":1}
I20260926 01:00:06.842505 11374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:06.837188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42006 (local address 127.3.75.3:46757)
0926 01:00:06.837372 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:06.837377 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:06.837388 (+ 11us) server_negotiation.cc:407] Connection header received
0926 01:00:06.837425 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:06.837427 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:06.837443 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:06.837690 (+ 247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:00:06.837746 (+ 56us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:06.838856 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:06.841261 (+ 2405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:06.842028 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:06.842143 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:06.842222 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:06.842343 (+ 121us) server_negotiation.cc:299] Negotiation successful
0926 01:00:06.842374 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":88,"threads_started":1}
I20260926 01:00:06.843212 11281 tablet_service.cc:1467] Processing CreateTablet for tablet 92c2a7441b7947c191ff9b041f4181f0 (DEFAULT_TABLE table=test-table [id=6f90b9c491f447a7a113574d4551c7bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:00:06.843565 11281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92c2a7441b7947c191ff9b041f4181f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:00:06.843528 11141 tablet_service.cc:1467] Processing CreateTablet for tablet 92c2a7441b7947c191ff9b041f4181f0 (DEFAULT_TABLE table=test-table [id=6f90b9c491f447a7a113574d4551c7bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:00:06.843833 11141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92c2a7441b7947c191ff9b041f4181f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:00:06.846464 11375 tablet_bootstrap.cc:492] T 92c2a7441b7947c191ff9b041f4181f0 P 78b75a2199c0452f9bbac4eb3746b057: Bootstrap starting.
I20260926 01:00:06.846812 11376 tablet_bootstrap.cc:492] T 92c2a7441b7947c191ff9b041f4181f0 P a42f67fb6fef43c6867b287287170a1f: Bootstrap starting.
I20260926 01:00:06.847342 11377 tablet_bootstrap.cc:492] T 92c2a7441b7947c191ff9b041f4181f0 P 47a2277a9fd24c14b23f901d6823409f: Bootstrap starting.
I20260926 01:00:06.847498 11375 tablet_bootstrap.cc:654] T 92c2a7441b7947c191ff9b041f4181f0 P 78b75a2199c0452f9bbac4eb3746b057: Neither blocks nor log segments found. Creating new log.
I20260926 01:00:06.847524 11376 tablet_bootstrap.cc:654] T 92c2a7441b7947c191ff9b041f4181f0 P a42f67fb6fef43c6867b287287170a1f: Neither blocks nor log segments found. Creating new log.
I20260926 01:00:06.847834 11376 log.cc:824] T 92c2a7441b7947c191ff9b041f4181f0 P a42f67fb6fef43c6867b287287170a1f: Log is configured to *not* fsync() on all Append() calls
I20260926 01:00:06.847962 11377 tablet_bootstrap.cc:654] T 92c2a7441b7947c191ff9b041f4181f0 P 47a2277a9fd24c14b23f901d6823409f: Neither blocks nor log segments found. Creating new log.
I20260926 01:00:06.848130 11375 log.cc:824] T 92c2a7441b7947c191ff9b041f4181f0 P 78b75a2199c0452f9bbac4eb3746b057: Log is configured to *not* fsync() on all Append() calls
I20260926 01:00:06.848289 11377 log.cc:824] T 92c2a7441b7947c191ff9b041f4181f0 P 47a2277a9fd24c14b23f901d6823409f: Log is configured to *not* fsync() on all Append() calls
I20260926 01:00:06.848579 11376 tablet_bootstrap.cc:492] T 92c2a7441b7947c191ff9b041f4181f0 P a42f67fb6fef43c6867b287287170a1f: No bootstrap required, opened a new log
I20260926 01:00:06.848645 11376 ts_tablet_manager.cc:1397] T 92c2a7441b7947c191ff9b041f4181f0 P a42f67fb6fef43c6867b287287170a1f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 01:00:06.848855 11375 tablet_bootstrap.cc:492] T 92c2a7441b7947c191ff9b041f4181f0 P 78b75a2199c0452f9bbac4eb3746b057: No bootstrap required, opened a new log
I20260926 01:00:06.848912 11375 ts_tablet_manager.cc:1397] T 92c2a7441b7947c191ff9b041f4181f0 P 78b75a2199c0452f9bbac4eb3746b057: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 01:00:06.848930 11377 tablet_bootstrap.cc:492] T 92c2a7441b7947c191ff9b041f4181f0 P 47a2277a9fd24c14b23f901d6823409f: No bootstrap required, opened a new log
I20260926 01:00:06.848973 11377 ts_tablet_manager.cc:1397] T 92c2a7441b7947c191ff9b041f4181f0 P 47a2277a9fd24c14b23f901d6823409f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:00:06.850476 11377 raft_consensus.cc:348] T 92c2a7441b7947c191ff9b041f4181f0 P 47a2277a9fd24c14b23f901d6823409f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a2277a9fd24c14b23f901d6823409f" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46757 } } peers { permanent_uuid: "78b75a2199c0452f9bbac4eb3746b057" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 45301 } } peers { permanent_uuid: "a42f67fb6fef43c6867b287287170a1f" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39043 } }
I20260926 01:00:06.850587 11377 raft_consensus.cc:374] T 92c2a7441b7947c191ff9b041f4181f0 P 47a2277a9fd24c14b23f901d6823409f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:00:06.850606 11377 raft_consensus.cc:729] T 92c2a7441b7947c191ff9b041f4181f0 P 47a2277a9fd24c14b23f901d6823409f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47a2277a9fd24c14b23f901d6823409f, State: Initialized, Role: FOLLOWER
I20260926 01:00:06.850747 11377 consensus_queue.cc:260] T 92c2a7441b7947c191ff9b041f4181f0 P 47a2277a9fd24c14b23f901d6823409f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a2277a9fd24c14b23f901d6823409f" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46757 } } peers { permanent_uuid: "78b75a2199c0452f9bbac4eb3746b057" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 45301 } } peers { permanent_uuid: "a42f67fb6fef43c6867b287287170a1f" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39043 } }
I20260926 01:00:06.850709 11376 raft_consensus.cc:348] T 92c2a7441b7947c191ff9b041f4181f0 P a42f67fb6fef43c6867b287287170a1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a2277a9fd24c14b23f901d6823409f" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46757 } } peers { permanent_uuid: "78b75a2199c0452f9bbac4eb3746b057" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 45301 } } peers { permanent_uuid: "a42f67fb6fef43c6867b287287170a1f" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39043 } }
I20260926 01:00:06.850864 11376 raft_consensus.cc:374] T 92c2a7441b7947c191ff9b041f4181f0 P a42f67fb6fef43c6867b287287170a1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:00:06.850802 11375 raft_consensus.cc:348] T 92c2a7441b7947c191ff9b041f4181f0 P 78b75a2199c0452f9bbac4eb3746b057 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a2277a9fd24c14b23f901d6823409f" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46757 } } peers { permanent_uuid: "78b75a2199c0452f9bbac4eb3746b057" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 45301 } } peers { permanent_uuid: "a42f67fb6fef43c6867b287287170a1f" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39043 } }
I20260926 01:00:06.850953 11376 raft_consensus.cc:729] T 92c2a7441b7947c191ff9b041f4181f0 P a42f67fb6fef43c6867b287287170a1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a42f67fb6fef43c6867b287287170a1f, State: Initialized, Role: FOLLOWER
I20260926 01:00:06.850978 11375 raft_consensus.cc:374] T 92c2a7441b7947c191ff9b041f4181f0 P 78b75a2199c0452f9bbac4eb3746b057 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:00:06.851022 11375 raft_consensus.cc:729] T 92c2a7441b7947c191ff9b041f4181f0 P 78b75a2199c0452f9bbac4eb3746b057 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b75a2199c0452f9bbac4eb3746b057, State: Initialized, Role: FOLLOWER
I20260926 01:00:06.851049 11347 heartbeater.cc:503] Master 127.3.75.62:33509 was elected leader, sending a full tablet report...
I20260926 01:00:06.851092 11376 consensus_queue.cc:260] T 92c2a7441b7947c191ff9b041f4181f0 P a42f67fb6fef43c6867b287287170a1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a2277a9fd24c14b23f901d6823409f" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46757 } } peers { permanent_uuid: "78b75a2199c0452f9bbac4eb3746b057" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 45301 } } peers { permanent_uuid: "a42f67fb6fef43c6867b287287170a1f" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39043 } }
I20260926 01:00:06.851110 11375 consensus_queue.cc:260] T 92c2a7441b7947c191ff9b041f4181f0 P 78b75a2199c0452f9bbac4eb3746b057 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a2277a9fd24c14b23f901d6823409f" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46757 } } peers { permanent_uuid: "78b75a2199c0452f9bbac4eb3746b057" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 45301 } } peers { permanent_uuid: "a42f67fb6fef43c6867b287287170a1f" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39043 } }
I20260926 01:00:06.851039 11377 ts_tablet_manager.cc:1428] T 92c2a7441b7947c191ff9b041f4181f0 P 47a2277a9fd24c14b23f901d6823409f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 01:00:06.851341 11376 ts_tablet_manager.cc:1428] T 92c2a7441b7947c191ff9b041f4181f0 P a42f67fb6fef43c6867b287287170a1f: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 01:00:06.851341 11375 ts_tablet_manager.cc:1428] T 92c2a7441b7947c191ff9b041f4181f0 P 78b75a2199c0452f9bbac4eb3746b057: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 01:00:06.851444 11207 heartbeater.cc:503] Master 127.3.75.62:33509 was elected leader, sending a full tablet report...
I20260926 01:00:06.851454 11067 heartbeater.cc:503] Master 127.3.75.62:33509 was elected leader, sending a full tablet report...
W20260926 01:00:06.880405 11208 tablet.cc:2369] T 92c2a7441b7947c191ff9b041f4181f0 P a42f67fb6fef43c6867b287287170a1f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:00:06.881412 11383 raft_consensus.cc:482] T 92c2a7441b7947c191ff9b041f4181f0 P a42f67fb6fef43c6867b287287170a1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:00:06.881500 11383 raft_consensus.cc:504] T 92c2a7441b7947c191ff9b041f4181f0 P a42f67fb6fef43c6867b287287170a1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a2277a9fd24c14b23f901d6823409f" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46757 } } peers { permanent_uuid: "78b75a2199c0452f9bbac4eb3746b057" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 45301 } } peers { permanent_uuid: "a42f67fb6fef43c6867b287287170a1f" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39043 } }
I20260926 01:00:06.881942 11383 leader_election.cc:288] T 92c2a7441b7947c191ff9b041f4181f0 P a42f67fb6fef43c6867b287287170a1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47a2277a9fd24c14b23f901d6823409f (127.3.75.3:46757), 78b75a2199c0452f9bbac4eb3746b057 (127.3.75.1:45301)
I20260926 01:00:06.884815 11374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:06.882199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:39565 (local address 127.3.75.3:46757)
0926 01:00:06.882264 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:06.882266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:06.882274 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:00:06.882300 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:06.882301 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:06.882311 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:06.882396 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:00:06.882558 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:06.883097 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:06.884087 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:06.884444 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:06.884535 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:06.884544 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:06.884704 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 01:00:06.884738 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 01:00:06.884846 11371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:06.882184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:40087 (local address 127.3.75.1:45301)
0926 01:00:06.882256 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:06.882258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:06.882426 (+ 168us) server_negotiation.cc:407] Connection header received
0926 01:00:06.882503 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:06.882505 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:06.882521 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:06.882591 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:00:06.882762 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:06.883226 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:06.884138 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:06.884485 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:06.884577 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:06.884586 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:06.884743 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 01:00:06.884777 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 01:00:06.884859 11100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:06.882122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.3:46757 (local address 127.3.75.2:39565)
0926 01:00:06.882183 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:06.882195 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:06.882236 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:06.882396 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:06.882397 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:06.882399 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:00:06.882472 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:06.882473 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:06.883181 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:06.883182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:06.883986 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:06.883989 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:06.884533 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:06.884534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:06.884614 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:06.884617 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:00:06.884698 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 01:00:06.884737 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":52}
I20260926 01:00:06.884871 11384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:06.882122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.1:45301 (local address 127.3.75.2:40087)
0926 01:00:06.882334 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:06.882346 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:06.882440 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:06.882594 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:06.882595 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:06.882596 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:00:06.882679 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:06.882681 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:06.883319 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:06.883320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:06.884038 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:06.884040 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:06.884578 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:06.884579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:06.884658 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:06.884661 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:00:06.884735 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 01:00:06.884775 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":29,"thread_start_us":122,"threads_started":1}
I20260926 01:00:06.885231 11301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92c2a7441b7947c191ff9b041f4181f0" candidate_uuid: "a42f67fb6fef43c6867b287287170a1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47a2277a9fd24c14b23f901d6823409f" is_pre_election: true
I20260926 01:00:06.885279 11021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92c2a7441b7947c191ff9b041f4181f0" candidate_uuid: "a42f67fb6fef43c6867b287287170a1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b75a2199c0452f9bbac4eb3746b057" is_pre_election: true
I20260926 01:00:06.885404 11301 raft_consensus.cc:2454] T 92c2a7441b7947c191ff9b041f4181f0 P 47a2277a9fd24c14b23f901d6823409f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a42f67fb6fef43c6867b287287170a1f in term 0.
I20260926 01:00:06.885426 11021 raft_consensus.cc:2454] T 92c2a7441b7947c191ff9b041f4181f0 P 78b75a2199c0452f9bbac4eb3746b057 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a42f67fb6fef43c6867b287287170a1f in term 0.
I20260926 01:00:06.885627 11094 leader_election.cc:302] T 92c2a7441b7947c191ff9b041f4181f0 P a42f67fb6fef43c6867b287287170a1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47a2277a9fd24c14b23f901d6823409f, a42f67fb6fef43c6867b287287170a1f; no voters:
I20260926 01:00:06.885767 11383 raft_consensus.cc:2781] T 92c2a7441b7947c191ff9b041f4181f0 P a42f67fb6fef43c6867b287287170a1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:00:06.885825 11383 raft_consensus.cc:482] T 92c2a7441b7947c191ff9b041f4181f0 P a42f67fb6fef43c6867b287287170a1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:00:06.885847 11383 raft_consensus.cc:3037] T 92c2a7441b7947c191ff9b041f4181f0 P a42f67fb6fef43c6867b287287170a1f [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:00:06.886726 11383 raft_consensus.cc:504] T 92c2a7441b7947c191ff9b041f4181f0 P a42f67fb6fef43c6867b287287170a1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a2277a9fd24c14b23f901d6823409f" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46757 } } peers { permanent_uuid: "78b75a2199c0452f9bbac4eb3746b057" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 45301 } } peers { permanent_uuid: "a42f67fb6fef43c6867b287287170a1f" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39043 } }
I20260926 01:00:06.886909 11383 leader_election.cc:288] T 92c2a7441b7947c191ff9b041f4181f0 P a42f67fb6fef43c6867b287287170a1f [CANDIDATE]: Term 1 election: Requested vote from peers 47a2277a9fd24c14b23f901d6823409f (127.3.75.3:46757), 78b75a2199c0452f9bbac4eb3746b057 (127.3.75.1:45301)
I20260926 01:00:06.887051 11301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92c2a7441b7947c191ff9b041f4181f0" candidate_uuid: "a42f67fb6fef43c6867b287287170a1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47a2277a9fd24c14b23f901d6823409f"
I20260926 01:00:06.887063 11021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92c2a7441b7947c191ff9b041f4181f0" candidate_uuid: "a42f67fb6fef43c6867b287287170a1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b75a2199c0452f9bbac4eb3746b057"
I20260926 01:00:06.887154 11301 raft_consensus.cc:3037] T 92c2a7441b7947c191ff9b041f4181f0 P 47a2277a9fd24c14b23f901d6823409f [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:00:06.887151 11021 raft_consensus.cc:3037] T 92c2a7441b7947c191ff9b041f4181f0 P 78b75a2199c0452f9bbac4eb3746b057 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:00:06.887827 11301 raft_consensus.cc:2454] T 92c2a7441b7947c191ff9b041f4181f0 P 47a2277a9fd24c14b23f901d6823409f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a42f67fb6fef43c6867b287287170a1f in term 1.
I20260926 01:00:06.887876 11021 raft_consensus.cc:2454] T 92c2a7441b7947c191ff9b041f4181f0 P 78b75a2199c0452f9bbac4eb3746b057 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a42f67fb6fef43c6867b287287170a1f in term 1.
I20260926 01:00:06.888072 11095 leader_election.cc:302] T 92c2a7441b7947c191ff9b041f4181f0 P a42f67fb6fef43c6867b287287170a1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b75a2199c0452f9bbac4eb3746b057, a42f67fb6fef43c6867b287287170a1f; no voters:
I20260926 01:00:06.888187 11383 raft_consensus.cc:2781] T 92c2a7441b7947c191ff9b041f4181f0 P a42f67fb6fef43c6867b287287170a1f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:00:06.888350 11383 raft_consensus.cc:686] T 92c2a7441b7947c191ff9b041f4181f0 P a42f67fb6fef43c6867b287287170a1f [term 1 LEADER]: Becoming Leader. State: Replica: a42f67fb6fef43c6867b287287170a1f, State: Running, Role: LEADER
I20260926 01:00:06.888463 11383 consensus_queue.cc:237] T 92c2a7441b7947c191ff9b041f4181f0 P a42f67fb6fef43c6867b287287170a1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a2277a9fd24c14b23f901d6823409f" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46757 } } peers { permanent_uuid: "78b75a2199c0452f9bbac4eb3746b057" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 45301 } } peers { permanent_uuid: "a42f67fb6fef43c6867b287287170a1f" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39043 } }
I20260926 01:00:06.889374 10864 catalog_manager.cc:5495] T 92c2a7441b7947c191ff9b041f4181f0 P a42f67fb6fef43c6867b287287170a1f reported cstate change: term changed from 0 to 1, leader changed from <none> to a42f67fb6fef43c6867b287287170a1f (127.3.75.2). New cstate: current_term: 1 leader_uuid: "a42f67fb6fef43c6867b287287170a1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a2277a9fd24c14b23f901d6823409f" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b75a2199c0452f9bbac4eb3746b057" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 45301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a42f67fb6fef43c6867b287287170a1f" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39043 } health_report { overall_health: HEALTHY } } }
I20260926 01:00:06.898545 11373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:06.896070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57100 (local address 127.3.75.2:39043)
0926 01:00:06.896144 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:06.896146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:06.896156 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:00:06.896223 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:06.896224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:06.896244 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:06.896348 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:00:06.896482 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:06.896952 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:06.897720 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:06.897933 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:06.898016 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:06.898193 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:00:06.898307 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:00:06.898400 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:06.898442 (+ 42us) server_negotiation.cc:299] Negotiation successful
0926 01:00:06.898462 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 01:00:06.900409 11301 raft_consensus.cc:1261] T 92c2a7441b7947c191ff9b041f4181f0 P 47a2277a9fd24c14b23f901d6823409f [term 1 FOLLOWER]: Refusing update from remote peer a42f67fb6fef43c6867b287287170a1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:00:06.900414 11021 raft_consensus.cc:1261] T 92c2a7441b7947c191ff9b041f4181f0 P 78b75a2199c0452f9bbac4eb3746b057 [term 1 FOLLOWER]: Refusing update from remote peer a42f67fb6fef43c6867b287287170a1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:00:06.900795 11385 consensus_queue.cc:1035] T 92c2a7441b7947c191ff9b041f4181f0 P a42f67fb6fef43c6867b287287170a1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b75a2199c0452f9bbac4eb3746b057" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 45301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:00:06.900952 11383 consensus_queue.cc:1035] T 92c2a7441b7947c191ff9b041f4181f0 P a42f67fb6fef43c6867b287287170a1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "47a2277a9fd24c14b23f901d6823409f" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:00:06.903759 11389 mvcc.cc:205] Tried to move back new op lower bound from 7333414530661068800 to 7333414530615922688. Current Snapshot: MvccSnapshot[applied={T|T < 7333414530661068800 or (T in {7333414530661068800})}]
I20260926 01:00:06.903923 11388 mvcc.cc:205] Tried to move back new op lower bound from 7333414530661068800 to 7333414530615922688. Current Snapshot: MvccSnapshot[applied={T|T < 7333414530661068800 or (T in {7333414530661068800})}]
I20260926 01:00:06.904173 11390 mvcc.cc:205] Tried to move back new op lower bound from 7333414530661068800 to 7333414530615922688. Current Snapshot: MvccSnapshot[applied={T|T < 7333414530661068800}]
I20260926 01:00:06.905567 10864 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:00:06.905654 10864 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:00:06.906425 10864 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 246badea3f0f4badadc5380fb63e9019: Sending DeleteTablet for 3 replicas of tablet 92c2a7441b7947c191ff9b041f4181f0
I20260926 01:00:06.906781 11141 tablet_service.cc:1514] Processing DeleteTablet for tablet 92c2a7441b7947c191ff9b041f4181f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:00:06 UTC) from {username='oryx', principal='oryx/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:57088
I20260926 01:00:06.906813 11281 tablet_service.cc:1514] Processing DeleteTablet for tablet 92c2a7441b7947c191ff9b041f4181f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:00:06 UTC) from {username='oryx', principal='oryx/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:42006
I20260926 01:00:06.906868 10864 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59038:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:00:06.907030 10864 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:00:06.907069 11001 tablet_service.cc:1514] Processing DeleteTablet for tablet 92c2a7441b7947c191ff9b041f4181f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:00:06 UTC) from {username='oryx', principal='oryx/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:50308
I20260926 01:00:06.907369 11398 tablet_replica.cc:330] T 92c2a7441b7947c191ff9b041f4181f0 P a42f67fb6fef43c6867b287287170a1f: stopping tablet replica
I20260926 01:00:06.907481 11400 tablet_replica.cc:330] T 92c2a7441b7947c191ff9b041f4181f0 P 78b75a2199c0452f9bbac4eb3746b057: stopping tablet replica
I20260926 01:00:06.907500 11398 raft_consensus.cc:2229] T 92c2a7441b7947c191ff9b041f4181f0 P a42f67fb6fef43c6867b287287170a1f [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:00:06.907588 11400 raft_consensus.cc:2229] T 92c2a7441b7947c191ff9b041f4181f0 P 78b75a2199c0452f9bbac4eb3746b057 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:00:06.907703 11400 raft_consensus.cc:2258] T 92c2a7441b7947c191ff9b041f4181f0 P 78b75a2199c0452f9bbac4eb3746b057 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:00:06.908247 11281 tablet_service.cc:1467] Processing CreateTablet for tablet 7a2c1fb104af4c6c90d949a1a9cb697a (DEFAULT_TABLE table=test-table [id=ec2ce6a2677a434b86c8e856a3bd4e8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:00:06.908272 11141 tablet_service.cc:1467] Processing CreateTablet for tablet 7a2c1fb104af4c6c90d949a1a9cb697a (DEFAULT_TABLE table=test-table [id=ec2ce6a2677a434b86c8e856a3bd4e8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:00:06.908403 11281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a2c1fb104af4c6c90d949a1a9cb697a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:00:06.908463 11141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a2c1fb104af4c6c90d949a1a9cb697a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:00:06.909723 11001 tablet_service.cc:1467] Processing CreateTablet for tablet 7a2c1fb104af4c6c90d949a1a9cb697a (DEFAULT_TABLE table=test-table [id=ec2ce6a2677a434b86c8e856a3bd4e8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:00:06.909878 11001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a2c1fb104af4c6c90d949a1a9cb697a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:00:06.907969 11398 raft_consensus.cc:2258] T 92c2a7441b7947c191ff9b041f4181f0 P a42f67fb6fef43c6867b287287170a1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:00:06.910327 11398 ts_tablet_manager.cc:1905] T 92c2a7441b7947c191ff9b041f4181f0 P a42f67fb6fef43c6867b287287170a1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:00:06.910982 11376 tablet_bootstrap.cc:492] T 7a2c1fb104af4c6c90d949a1a9cb697a P a42f67fb6fef43c6867b287287170a1f: Bootstrap starting.
I20260926 01:00:06.911044 11375 tablet_bootstrap.cc:492] T 7a2c1fb104af4c6c90d949a1a9cb697a P 78b75a2199c0452f9bbac4eb3746b057: Bootstrap starting.
I20260926 01:00:06.911453 11376 tablet_bootstrap.cc:654] T 7a2c1fb104af4c6c90d949a1a9cb697a P a42f67fb6fef43c6867b287287170a1f: Neither blocks nor log segments found. Creating new log.
I20260926 01:00:06.911453 11375 tablet_bootstrap.cc:654] T 7a2c1fb104af4c6c90d949a1a9cb697a P 78b75a2199c0452f9bbac4eb3746b057: Neither blocks nor log segments found. Creating new log.
I20260926 01:00:06.911919 11398 ts_tablet_manager.cc:1918] T 92c2a7441b7947c191ff9b041f4181f0 P a42f67fb6fef43c6867b287287170a1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:00:06.911979 11398 log.cc:1196] T 92c2a7441b7947c191ff9b041f4181f0 P a42f67fb6fef43c6867b287287170a1f: Deleting WAL directory at /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/ts-1/wal/wals/92c2a7441b7947c191ff9b041f4181f0
I20260926 01:00:06.912114 11376 tablet_bootstrap.cc:492] T 7a2c1fb104af4c6c90d949a1a9cb697a P a42f67fb6fef43c6867b287287170a1f: No bootstrap required, opened a new log
I20260926 01:00:06.912160 11376 ts_tablet_manager.cc:1397] T 7a2c1fb104af4c6c90d949a1a9cb697a P a42f67fb6fef43c6867b287287170a1f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 01:00:06.912173 11377 tablet_bootstrap.cc:492] T 7a2c1fb104af4c6c90d949a1a9cb697a P 47a2277a9fd24c14b23f901d6823409f: Bootstrap starting.
I20260926 01:00:06.912304 11376 raft_consensus.cc:348] T 7a2c1fb104af4c6c90d949a1a9cb697a P a42f67fb6fef43c6867b287287170a1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b75a2199c0452f9bbac4eb3746b057" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 45301 } } peers { permanent_uuid: "a42f67fb6fef43c6867b287287170a1f" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39043 } } peers { permanent_uuid: "47a2277a9fd24c14b23f901d6823409f" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46757 } }
I20260926 01:00:06.912374 11376 raft_consensus.cc:374] T 7a2c1fb104af4c6c90d949a1a9cb697a P a42f67fb6fef43c6867b287287170a1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:00:06.912395 11376 raft_consensus.cc:729] T 7a2c1fb104af4c6c90d949a1a9cb697a P a42f67fb6fef43c6867b287287170a1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a42f67fb6fef43c6867b287287170a1f, State: Initialized, Role: FOLLOWER
I20260926 01:00:06.912437 11376 consensus_queue.cc:260] T 7a2c1fb104af4c6c90d949a1a9cb697a P a42f67fb6fef43c6867b287287170a1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b75a2199c0452f9bbac4eb3746b057" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 45301 } } peers { permanent_uuid: "a42f67fb6fef43c6867b287287170a1f" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39043 } } peers { permanent_uuid: "47a2277a9fd24c14b23f901d6823409f" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46757 } }
I20260926 01:00:06.912549 11376 ts_tablet_manager.cc:1428] T 7a2c1fb104af4c6c90d949a1a9cb697a P a42f67fb6fef43c6867b287287170a1f: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 01:00:06.912581 11377 tablet_bootstrap.cc:654] T 7a2c1fb104af4c6c90d949a1a9cb697a P 47a2277a9fd24c14b23f901d6823409f: Neither blocks nor log segments found. Creating new log.
I20260926 01:00:06.912850 11400 ts_tablet_manager.cc:1905] T 92c2a7441b7947c191ff9b041f4181f0 P 78b75a2199c0452f9bbac4eb3746b057: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:00:06.913074 11375 tablet_bootstrap.cc:492] T 7a2c1fb104af4c6c90d949a1a9cb697a P 78b75a2199c0452f9bbac4eb3746b057: No bootstrap required, opened a new log
I20260926 01:00:06.913123 11375 ts_tablet_manager.cc:1397] T 7a2c1fb104af4c6c90d949a1a9cb697a P 78b75a2199c0452f9bbac4eb3746b057: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:00:06.913269 11375 raft_consensus.cc:348] T 7a2c1fb104af4c6c90d949a1a9cb697a P 78b75a2199c0452f9bbac4eb3746b057 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b75a2199c0452f9bbac4eb3746b057" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 45301 } } peers { permanent_uuid: "a42f67fb6fef43c6867b287287170a1f" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39043 } } peers { permanent_uuid: "47a2277a9fd24c14b23f901d6823409f" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46757 } }
I20260926 01:00:06.913349 11375 raft_consensus.cc:374] T 7a2c1fb104af4c6c90d949a1a9cb697a P 78b75a2199c0452f9bbac4eb3746b057 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:00:06.913378 11375 raft_consensus.cc:729] T 7a2c1fb104af4c6c90d949a1a9cb697a P 78b75a2199c0452f9bbac4eb3746b057 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b75a2199c0452f9bbac4eb3746b057, State: Initialized, Role: FOLLOWER
I20260926 01:00:06.913421 11375 consensus_queue.cc:260] T 7a2c1fb104af4c6c90d949a1a9cb697a P 78b75a2199c0452f9bbac4eb3746b057 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b75a2199c0452f9bbac4eb3746b057" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 45301 } } peers { permanent_uuid: "a42f67fb6fef43c6867b287287170a1f" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39043 } } peers { permanent_uuid: "47a2277a9fd24c14b23f901d6823409f" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46757 } }
I20260926 01:00:06.913508 11375 ts_tablet_manager.cc:1428] T 7a2c1fb104af4c6c90d949a1a9cb697a P 78b75a2199c0452f9bbac4eb3746b057: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 01:00:06.913568 11398 ts_tablet_manager.cc:1939] T 92c2a7441b7947c191ff9b041f4181f0 P a42f67fb6fef43c6867b287287170a1f: Deleting consensus metadata
I20260926 01:00:06.913945 10851 catalog_manager.cc:4841] TS a42f67fb6fef43c6867b287287170a1f (127.3.75.2:39043): tablet 92c2a7441b7947c191ff9b041f4181f0 (table test-table [id=6f90b9c491f447a7a113574d4551c7bc]) successfully deleted
I20260926 01:00:06.914372 11377 tablet_bootstrap.cc:492] T 7a2c1fb104af4c6c90d949a1a9cb697a P 47a2277a9fd24c14b23f901d6823409f: No bootstrap required, opened a new log
I20260926 01:00:06.914427 11377 ts_tablet_manager.cc:1397] T 7a2c1fb104af4c6c90d949a1a9cb697a P 47a2277a9fd24c14b23f901d6823409f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 01:00:06.914574 11377 raft_consensus.cc:348] T 7a2c1fb104af4c6c90d949a1a9cb697a P 47a2277a9fd24c14b23f901d6823409f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b75a2199c0452f9bbac4eb3746b057" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 45301 } } peers { permanent_uuid: "a42f67fb6fef43c6867b287287170a1f" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39043 } } peers { permanent_uuid: "47a2277a9fd24c14b23f901d6823409f" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46757 } }
I20260926 01:00:06.914645 11377 raft_consensus.cc:374] T 7a2c1fb104af4c6c90d949a1a9cb697a P 47a2277a9fd24c14b23f901d6823409f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:00:06.914666 11377 raft_consensus.cc:729] T 7a2c1fb104af4c6c90d949a1a9cb697a P 47a2277a9fd24c14b23f901d6823409f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47a2277a9fd24c14b23f901d6823409f, State: Initialized, Role: FOLLOWER
I20260926 01:00:06.914707 11377 consensus_queue.cc:260] T 7a2c1fb104af4c6c90d949a1a9cb697a P 47a2277a9fd24c14b23f901d6823409f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b75a2199c0452f9bbac4eb3746b057" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 45301 } } peers { permanent_uuid: "a42f67fb6fef43c6867b287287170a1f" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39043 } } peers { permanent_uuid: "47a2277a9fd24c14b23f901d6823409f" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46757 } }
I20260926 01:00:06.914790 11377 ts_tablet_manager.cc:1428] T 7a2c1fb104af4c6c90d949a1a9cb697a P 47a2277a9fd24c14b23f901d6823409f: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 01:00:06.914901 11400 ts_tablet_manager.cc:1918] T 92c2a7441b7947c191ff9b041f4181f0 P 78b75a2199c0452f9bbac4eb3746b057: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:00:06.914971 11400 log.cc:1196] T 92c2a7441b7947c191ff9b041f4181f0 P 78b75a2199c0452f9bbac4eb3746b057: Deleting WAL directory at /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/ts-0/wal/wals/92c2a7441b7947c191ff9b041f4181f0
I20260926 01:00:06.915246 11400 ts_tablet_manager.cc:1939] T 92c2a7441b7947c191ff9b041f4181f0 P 78b75a2199c0452f9bbac4eb3746b057: Deleting consensus metadata
I20260926 01:00:06.915728 10851 catalog_manager.cc:4841] TS 78b75a2199c0452f9bbac4eb3746b057 (127.3.75.1:45301): tablet 92c2a7441b7947c191ff9b041f4181f0 (table test-table [id=6f90b9c491f447a7a113574d4551c7bc]) successfully deleted
I20260926 01:00:06.919000 11399 tablet_replica.cc:330] T 92c2a7441b7947c191ff9b041f4181f0 P 47a2277a9fd24c14b23f901d6823409f: stopping tablet replica
I20260926 01:00:06.919129 11399 raft_consensus.cc:2229] T 92c2a7441b7947c191ff9b041f4181f0 P 47a2277a9fd24c14b23f901d6823409f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:00:06.919277 11399 raft_consensus.cc:2258] T 92c2a7441b7947c191ff9b041f4181f0 P 47a2277a9fd24c14b23f901d6823409f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:00:06.919731 11399 ts_tablet_manager.cc:1905] T 92c2a7441b7947c191ff9b041f4181f0 P 47a2277a9fd24c14b23f901d6823409f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:00:06.921382 11399 ts_tablet_manager.cc:1918] T 92c2a7441b7947c191ff9b041f4181f0 P 47a2277a9fd24c14b23f901d6823409f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:00:06.921473 11399 log.cc:1196] T 92c2a7441b7947c191ff9b041f4181f0 P 47a2277a9fd24c14b23f901d6823409f: Deleting WAL directory at /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/ts-2/wal/wals/92c2a7441b7947c191ff9b041f4181f0
I20260926 01:00:06.921708 11399 ts_tablet_manager.cc:1939] T 92c2a7441b7947c191ff9b041f4181f0 P 47a2277a9fd24c14b23f901d6823409f: Deleting consensus metadata
I20260926 01:00:06.922117 10850 catalog_manager.cc:4841] TS 47a2277a9fd24c14b23f901d6823409f (127.3.75.3:46757): tablet 92c2a7441b7947c191ff9b041f4181f0 (table test-table [id=6f90b9c491f447a7a113574d4551c7bc]) successfully deleted
W20260926 01:00:06.962168 11068 tablet.cc:2369] T 7a2c1fb104af4c6c90d949a1a9cb697a P 78b75a2199c0452f9bbac4eb3746b057: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 01:00:07.047147 11348 tablet.cc:2369] T 7a2c1fb104af4c6c90d949a1a9cb697a P 47a2277a9fd24c14b23f901d6823409f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:00:07.131655 11382 raft_consensus.cc:482] T 7a2c1fb104af4c6c90d949a1a9cb697a P 78b75a2199c0452f9bbac4eb3746b057 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:00:07.131789 11382 raft_consensus.cc:504] T 7a2c1fb104af4c6c90d949a1a9cb697a P 78b75a2199c0452f9bbac4eb3746b057 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b75a2199c0452f9bbac4eb3746b057" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 45301 } } peers { permanent_uuid: "a42f67fb6fef43c6867b287287170a1f" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39043 } } peers { permanent_uuid: "47a2277a9fd24c14b23f901d6823409f" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46757 } }
I20260926 01:00:07.132164 11382 leader_election.cc:288] T 7a2c1fb104af4c6c90d949a1a9cb697a P 78b75a2199c0452f9bbac4eb3746b057 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a42f67fb6fef43c6867b287287170a1f (127.3.75.2:39043), 47a2277a9fd24c14b23f901d6823409f (127.3.75.3:46757)
I20260926 01:00:07.135370 11374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:07.132476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:37157 (local address 127.3.75.3:46757)
0926 01:00:07.132551 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:07.132553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:07.132603 (+ 50us) server_negotiation.cc:407] Connection header received
0926 01:00:07.132776 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:07.132778 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:07.132788 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:07.132861 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:00:07.133018 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:07.133501 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:07.134475 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:07.135008 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:07.135092 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:07.135102 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:07.135246 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 01:00:07.135279 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 01:00:07.135370 11405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:07.132357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.3:46757 (local address 127.3.75.1:37157)
0926 01:00:07.132511 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:07.132522 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:07.132623 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:07.132864 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:07.132865 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:07.132867 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:00:07.132946 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:07.132948 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:07.133588 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:07.133589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:07.134380 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:07.134382 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:07.135103 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:07.135104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:07.135179 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:07.135183 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:00:07.135232 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 01:00:07.135277 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":32,"thread_start_us":65,"threads_started":1}
I20260926 01:00:07.135435 11373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:07.132376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:39519 (local address 127.3.75.2:39043)
0926 01:00:07.132431 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:07.132434 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:07.132537 (+ 103us) server_negotiation.cc:407] Connection header received
0926 01:00:07.132640 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:07.132641 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:07.132651 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:07.132708 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:00:07.132930 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:07.133443 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:07.134519 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:07.135106 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:07.135172 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:07.135181 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:07.135326 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 01:00:07.135364 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 01:00:07.135527 11404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:07.132326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.2:39043 (local address 127.3.75.1:39519)
0926 01:00:07.132457 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:07.132469 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:07.132543 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:07.132761 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:07.132763 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:07.132765 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:00:07.132844 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:07.132846 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:07.133527 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:07.133528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:07.134387 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:07.134389 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:07.135185 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:07.135186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:07.135252 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:07.135254 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 01:00:07.135305 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 01:00:07.135335 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":49,"thread_start_us":81,"threads_started":1}
I20260926 01:00:07.135774 11161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2c1fb104af4c6c90d949a1a9cb697a" candidate_uuid: "78b75a2199c0452f9bbac4eb3746b057" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a42f67fb6fef43c6867b287287170a1f" is_pre_election: true
I20260926 01:00:07.135854 11301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2c1fb104af4c6c90d949a1a9cb697a" candidate_uuid: "78b75a2199c0452f9bbac4eb3746b057" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47a2277a9fd24c14b23f901d6823409f" is_pre_election: true
I20260926 01:00:07.135924 11301 raft_consensus.cc:2454] T 7a2c1fb104af4c6c90d949a1a9cb697a P 47a2277a9fd24c14b23f901d6823409f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b75a2199c0452f9bbac4eb3746b057 in term 0.
I20260926 01:00:07.135939 11161 raft_consensus.cc:2454] T 7a2c1fb104af4c6c90d949a1a9cb697a P a42f67fb6fef43c6867b287287170a1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b75a2199c0452f9bbac4eb3746b057 in term 0.
I20260926 01:00:07.136145 10954 leader_election.cc:302] T 7a2c1fb104af4c6c90d949a1a9cb697a P 78b75a2199c0452f9bbac4eb3746b057 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47a2277a9fd24c14b23f901d6823409f, 78b75a2199c0452f9bbac4eb3746b057; no voters:
I20260926 01:00:07.136303 11382 raft_consensus.cc:2781] T 7a2c1fb104af4c6c90d949a1a9cb697a P 78b75a2199c0452f9bbac4eb3746b057 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:00:07.136356 11382 raft_consensus.cc:482] T 7a2c1fb104af4c6c90d949a1a9cb697a P 78b75a2199c0452f9bbac4eb3746b057 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:00:07.136381 11382 raft_consensus.cc:3037] T 7a2c1fb104af4c6c90d949a1a9cb697a P 78b75a2199c0452f9bbac4eb3746b057 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:00:07.137048 11382 raft_consensus.cc:504] T 7a2c1fb104af4c6c90d949a1a9cb697a P 78b75a2199c0452f9bbac4eb3746b057 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b75a2199c0452f9bbac4eb3746b057" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 45301 } } peers { permanent_uuid: "a42f67fb6fef43c6867b287287170a1f" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39043 } } peers { permanent_uuid: "47a2277a9fd24c14b23f901d6823409f" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46757 } }
I20260926 01:00:07.137203 11382 leader_election.cc:288] T 7a2c1fb104af4c6c90d949a1a9cb697a P 78b75a2199c0452f9bbac4eb3746b057 [CANDIDATE]: Term 1 election: Requested vote from peers a42f67fb6fef43c6867b287287170a1f (127.3.75.2:39043), 47a2277a9fd24c14b23f901d6823409f (127.3.75.3:46757)
I20260926 01:00:07.137394 11301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2c1fb104af4c6c90d949a1a9cb697a" candidate_uuid: "78b75a2199c0452f9bbac4eb3746b057" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47a2277a9fd24c14b23f901d6823409f"
I20260926 01:00:07.137467 11301 raft_consensus.cc:3037] T 7a2c1fb104af4c6c90d949a1a9cb697a P 47a2277a9fd24c14b23f901d6823409f [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:00:07.138100 11301 raft_consensus.cc:2454] T 7a2c1fb104af4c6c90d949a1a9cb697a P 47a2277a9fd24c14b23f901d6823409f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b75a2199c0452f9bbac4eb3746b057 in term 1.
I20260926 01:00:07.138244 11161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2c1fb104af4c6c90d949a1a9cb697a" candidate_uuid: "78b75a2199c0452f9bbac4eb3746b057" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a42f67fb6fef43c6867b287287170a1f"
I20260926 01:00:07.138291 10954 leader_election.cc:302] T 7a2c1fb104af4c6c90d949a1a9cb697a P 78b75a2199c0452f9bbac4eb3746b057 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47a2277a9fd24c14b23f901d6823409f, 78b75a2199c0452f9bbac4eb3746b057; no voters:
I20260926 01:00:07.138319 11161 raft_consensus.cc:3037] T 7a2c1fb104af4c6c90d949a1a9cb697a P a42f67fb6fef43c6867b287287170a1f [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:00:07.138401 11382 raft_consensus.cc:2781] T 7a2c1fb104af4c6c90d949a1a9cb697a P 78b75a2199c0452f9bbac4eb3746b057 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:00:07.138520 11382 raft_consensus.cc:686] T 7a2c1fb104af4c6c90d949a1a9cb697a P 78b75a2199c0452f9bbac4eb3746b057 [term 1 LEADER]: Becoming Leader. State: Replica: 78b75a2199c0452f9bbac4eb3746b057, State: Running, Role: LEADER
I20260926 01:00:07.138597 11382 consensus_queue.cc:237] T 7a2c1fb104af4c6c90d949a1a9cb697a P 78b75a2199c0452f9bbac4eb3746b057 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b75a2199c0452f9bbac4eb3746b057" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 45301 } } peers { permanent_uuid: "a42f67fb6fef43c6867b287287170a1f" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39043 } } peers { permanent_uuid: "47a2277a9fd24c14b23f901d6823409f" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46757 } }
I20260926 01:00:07.139196 11161 raft_consensus.cc:2454] T 7a2c1fb104af4c6c90d949a1a9cb697a P a42f67fb6fef43c6867b287287170a1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b75a2199c0452f9bbac4eb3746b057 in term 1.
I20260926 01:00:07.139241 10864 catalog_manager.cc:5495] T 7a2c1fb104af4c6c90d949a1a9cb697a P 78b75a2199c0452f9bbac4eb3746b057 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78b75a2199c0452f9bbac4eb3746b057 (127.3.75.1). New cstate: current_term: 1 leader_uuid: "78b75a2199c0452f9bbac4eb3746b057" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b75a2199c0452f9bbac4eb3746b057" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 45301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a42f67fb6fef43c6867b287287170a1f" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47a2277a9fd24c14b23f901d6823409f" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46757 } health_report { overall_health: UNKNOWN } } }
Sep 26 01:00:07 dist-test-slave-7v10 krb5kdc[10819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384406, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.75.62@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM
I20260926 01:00:07.214919 10913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:07.207817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59050 (local address 127.3.75.62:33509)
0926 01:00:07.207900 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:07.207903 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:07.208022 (+ 119us) server_negotiation.cc:407] Connection header received
0926 01:00:07.208111 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:07.208114 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:07.208129 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:07.208201 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:07.208683 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:07.209171 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:07.210004 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:07.210281 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:07.210350 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:07.212420 (+ 2070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:07.212422 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:07.212422 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:00:07.212429 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:07.213671 (+ 1242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:07.214068 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:07.214068 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:07.214069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:07.214127 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:07.214333 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:07.214334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:07.214335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:07.214448 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:07.214528 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:07.214687 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 01:00:07.214773 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 01:00:07.214986 11413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:07.207694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:33509 (local address 127.0.0.1:59050)
0926 01:00:07.207929 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:07.207940 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:07.208041 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:07.208204 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:07.208205 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:07.208539 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:07.208589 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:07.208591 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:07.209303 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:07.209304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:07.209901 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:07.209903 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:07.210365 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:07.210366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:07.210422 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:07.211001 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:07.211005 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:07.212321 (+ 1316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:00:07.213775 (+ 1454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:07.213782 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:07.213785 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:07.213973 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:07.214198 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:07.214199 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:07.214199 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:07.214263 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:07.214519 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:00:07.214526 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:00:07.214572 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 01:00:07.214663 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 01:00:07.214829 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"mutex_wait_us":68,"thread_start_us":132,"threads_started":1}
I20260926 01:00:07.217190 10864 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:59050:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 01:00:07.217438 10864 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:00:07.218922 11281 tablet_service.cc:1467] Processing CreateTablet for tablet bf333c7b63a04d6fb39f4811efcfeec0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ac79280e493b4c799c413e0eefbf77ce]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 01:00:07.218921 11001 tablet_service.cc:1467] Processing CreateTablet for tablet bf333c7b63a04d6fb39f4811efcfeec0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ac79280e493b4c799c413e0eefbf77ce]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 01:00:07.218982 11141 tablet_service.cc:1467] Processing CreateTablet for tablet bf333c7b63a04d6fb39f4811efcfeec0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ac79280e493b4c799c413e0eefbf77ce]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 01:00:07.219077 11281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf333c7b63a04d6fb39f4811efcfeec0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:00:07.219077 11001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf333c7b63a04d6fb39f4811efcfeec0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:00:07.219147 11141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf333c7b63a04d6fb39f4811efcfeec0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:00:07.220278 11376 tablet_bootstrap.cc:492] T bf333c7b63a04d6fb39f4811efcfeec0 P a42f67fb6fef43c6867b287287170a1f: Bootstrap starting.
I20260926 01:00:07.220527 11377 tablet_bootstrap.cc:492] T bf333c7b63a04d6fb39f4811efcfeec0 P 47a2277a9fd24c14b23f901d6823409f: Bootstrap starting.
I20260926 01:00:07.220527 11375 tablet_bootstrap.cc:492] T bf333c7b63a04d6fb39f4811efcfeec0 P 78b75a2199c0452f9bbac4eb3746b057: Bootstrap starting.
I20260926 01:00:07.220659 11376 tablet_bootstrap.cc:654] T bf333c7b63a04d6fb39f4811efcfeec0 P a42f67fb6fef43c6867b287287170a1f: Neither blocks nor log segments found. Creating new log.
I20260926 01:00:07.220961 11377 tablet_bootstrap.cc:654] T bf333c7b63a04d6fb39f4811efcfeec0 P 47a2277a9fd24c14b23f901d6823409f: Neither blocks nor log segments found. Creating new log.
I20260926 01:00:07.220963 11375 tablet_bootstrap.cc:654] T bf333c7b63a04d6fb39f4811efcfeec0 P 78b75a2199c0452f9bbac4eb3746b057: Neither blocks nor log segments found. Creating new log.
I20260926 01:00:07.221294 11376 tablet_bootstrap.cc:492] T bf333c7b63a04d6fb39f4811efcfeec0 P a42f67fb6fef43c6867b287287170a1f: No bootstrap required, opened a new log
I20260926 01:00:07.221346 11376 ts_tablet_manager.cc:1397] T bf333c7b63a04d6fb39f4811efcfeec0 P a42f67fb6fef43c6867b287287170a1f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 01:00:07.221455 11376 raft_consensus.cc:348] T bf333c7b63a04d6fb39f4811efcfeec0 P a42f67fb6fef43c6867b287287170a1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b75a2199c0452f9bbac4eb3746b057" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 45301 } } peers { permanent_uuid: "a42f67fb6fef43c6867b287287170a1f" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39043 } } peers { permanent_uuid: "47a2277a9fd24c14b23f901d6823409f" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46757 } }
I20260926 01:00:07.221534 11376 raft_consensus.cc:374] T bf333c7b63a04d6fb39f4811efcfeec0 P a42f67fb6fef43c6867b287287170a1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:00:07.221567 11376 raft_consensus.cc:729] T bf333c7b63a04d6fb39f4811efcfeec0 P a42f67fb6fef43c6867b287287170a1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a42f67fb6fef43c6867b287287170a1f, State: Initialized, Role: FOLLOWER
I20260926 01:00:07.221616 11377 tablet_bootstrap.cc:492] T bf333c7b63a04d6fb39f4811efcfeec0 P 47a2277a9fd24c14b23f901d6823409f: No bootstrap required, opened a new log
I20260926 01:00:07.221619 11376 consensus_queue.cc:260] T bf333c7b63a04d6fb39f4811efcfeec0 P a42f67fb6fef43c6867b287287170a1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b75a2199c0452f9bbac4eb3746b057" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 45301 } } peers { permanent_uuid: "a42f67fb6fef43c6867b287287170a1f" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39043 } } peers { permanent_uuid: "47a2277a9fd24c14b23f901d6823409f" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46757 } }
I20260926 01:00:07.221668 11377 ts_tablet_manager.cc:1397] T bf333c7b63a04d6fb39f4811efcfeec0 P 47a2277a9fd24c14b23f901d6823409f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 01:00:07.221709 11376 ts_tablet_manager.cc:1428] T bf333c7b63a04d6fb39f4811efcfeec0 P a42f67fb6fef43c6867b287287170a1f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:07.221810 11377 raft_consensus.cc:348] T bf333c7b63a04d6fb39f4811efcfeec0 P 47a2277a9fd24c14b23f901d6823409f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b75a2199c0452f9bbac4eb3746b057" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 45301 } } peers { permanent_uuid: "a42f67fb6fef43c6867b287287170a1f" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39043 } } peers { permanent_uuid: "47a2277a9fd24c14b23f901d6823409f" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46757 } }
I20260926 01:00:07.221899 11377 raft_consensus.cc:374] T bf333c7b63a04d6fb39f4811efcfeec0 P 47a2277a9fd24c14b23f901d6823409f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:00:07.221933 11377 raft_consensus.cc:729] T bf333c7b63a04d6fb39f4811efcfeec0 P 47a2277a9fd24c14b23f901d6823409f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47a2277a9fd24c14b23f901d6823409f, State: Initialized, Role: FOLLOWER
I20260926 01:00:07.221980 11377 consensus_queue.cc:260] T bf333c7b63a04d6fb39f4811efcfeec0 P 47a2277a9fd24c14b23f901d6823409f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b75a2199c0452f9bbac4eb3746b057" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 45301 } } peers { permanent_uuid: "a42f67fb6fef43c6867b287287170a1f" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39043 } } peers { permanent_uuid: "47a2277a9fd24c14b23f901d6823409f" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46757 } }
I20260926 01:00:07.222065 11377 ts_tablet_manager.cc:1428] T bf333c7b63a04d6fb39f4811efcfeec0 P 47a2277a9fd24c14b23f901d6823409f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:07.222117 11381 tablet_replica.cc:439] T bf333c7b63a04d6fb39f4811efcfeec0 P 47a2277a9fd24c14b23f901d6823409f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b75a2199c0452f9bbac4eb3746b057" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 45301 } } peers { permanent_uuid: "a42f67fb6fef43c6867b287287170a1f" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39043 } } peers { permanent_uuid: "47a2277a9fd24c14b23f901d6823409f" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46757 } } }
I20260926 01:00:07.222150 11395 tablet_replica.cc:439] T bf333c7b63a04d6fb39f4811efcfeec0 P a42f67fb6fef43c6867b287287170a1f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b75a2199c0452f9bbac4eb3746b057" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 45301 } } peers { permanent_uuid: "a42f67fb6fef43c6867b287287170a1f" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39043 } } peers { permanent_uuid: "47a2277a9fd24c14b23f901d6823409f" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46757 } } }
I20260926 01:00:07.222193 11381 tablet_replica.cc:442] T bf333c7b63a04d6fb39f4811efcfeec0 P 47a2277a9fd24c14b23f901d6823409f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 01:00:07.222225 11395 tablet_replica.cc:442] T bf333c7b63a04d6fb39f4811efcfeec0 P a42f67fb6fef43c6867b287287170a1f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 01:00:07.222718 11375 tablet_bootstrap.cc:492] T bf333c7b63a04d6fb39f4811efcfeec0 P 78b75a2199c0452f9bbac4eb3746b057: No bootstrap required, opened a new log
I20260926 01:00:07.222782 11375 ts_tablet_manager.cc:1397] T bf333c7b63a04d6fb39f4811efcfeec0 P 78b75a2199c0452f9bbac4eb3746b057: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:00:07.222947 11375 raft_consensus.cc:348] T bf333c7b63a04d6fb39f4811efcfeec0 P 78b75a2199c0452f9bbac4eb3746b057 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b75a2199c0452f9bbac4eb3746b057" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 45301 } } peers { permanent_uuid: "a42f67fb6fef43c6867b287287170a1f" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39043 } } peers { permanent_uuid: "47a2277a9fd24c14b23f901d6823409f" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46757 } }
I20260926 01:00:07.223016 11375 raft_consensus.cc:374] T bf333c7b63a04d6fb39f4811efcfeec0 P 78b75a2199c0452f9bbac4eb3746b057 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:00:07.223030 11375 raft_consensus.cc:729] T bf333c7b63a04d6fb39f4811efcfeec0 P 78b75a2199c0452f9bbac4eb3746b057 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b75a2199c0452f9bbac4eb3746b057, State: Initialized, Role: FOLLOWER
I20260926 01:00:07.223085 11375 consensus_queue.cc:260] T bf333c7b63a04d6fb39f4811efcfeec0 P 78b75a2199c0452f9bbac4eb3746b057 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b75a2199c0452f9bbac4eb3746b057" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 45301 } } peers { permanent_uuid: "a42f67fb6fef43c6867b287287170a1f" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39043 } } peers { permanent_uuid: "47a2277a9fd24c14b23f901d6823409f" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46757 } }
I20260926 01:00:07.223199 11375 ts_tablet_manager.cc:1428] T bf333c7b63a04d6fb39f4811efcfeec0 P 78b75a2199c0452f9bbac4eb3746b057: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:07.223268 11406 tablet_replica.cc:439] T bf333c7b63a04d6fb39f4811efcfeec0 P 78b75a2199c0452f9bbac4eb3746b057: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b75a2199c0452f9bbac4eb3746b057" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 45301 } } peers { permanent_uuid: "a42f67fb6fef43c6867b287287170a1f" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39043 } } peers { permanent_uuid: "47a2277a9fd24c14b23f901d6823409f" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46757 } } }
I20260926 01:00:07.223371 11406 tablet_replica.cc:442] T bf333c7b63a04d6fb39f4811efcfeec0 P 78b75a2199c0452f9bbac4eb3746b057: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 01:00:07.355440 11406 raft_consensus.cc:482] T bf333c7b63a04d6fb39f4811efcfeec0 P 78b75a2199c0452f9bbac4eb3746b057 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:00:07.355575 11406 raft_consensus.cc:504] T bf333c7b63a04d6fb39f4811efcfeec0 P 78b75a2199c0452f9bbac4eb3746b057 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b75a2199c0452f9bbac4eb3746b057" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 45301 } } peers { permanent_uuid: "a42f67fb6fef43c6867b287287170a1f" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39043 } } peers { permanent_uuid: "47a2277a9fd24c14b23f901d6823409f" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46757 } }
I20260926 01:00:07.355814 11406 leader_election.cc:288] T bf333c7b63a04d6fb39f4811efcfeec0 P 78b75a2199c0452f9bbac4eb3746b057 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a42f67fb6fef43c6867b287287170a1f (127.3.75.2:39043), 47a2277a9fd24c14b23f901d6823409f (127.3.75.3:46757)
I20260926 01:00:07.356070 11301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf333c7b63a04d6fb39f4811efcfeec0" candidate_uuid: "78b75a2199c0452f9bbac4eb3746b057" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47a2277a9fd24c14b23f901d6823409f" is_pre_election: true
I20260926 01:00:07.356076 11161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf333c7b63a04d6fb39f4811efcfeec0" candidate_uuid: "78b75a2199c0452f9bbac4eb3746b057" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a42f67fb6fef43c6867b287287170a1f" is_pre_election: true
I20260926 01:00:07.356179 11301 raft_consensus.cc:2454] T bf333c7b63a04d6fb39f4811efcfeec0 P 47a2277a9fd24c14b23f901d6823409f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b75a2199c0452f9bbac4eb3746b057 in term 0.
I20260926 01:00:07.356179 11161 raft_consensus.cc:2454] T bf333c7b63a04d6fb39f4811efcfeec0 P a42f67fb6fef43c6867b287287170a1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b75a2199c0452f9bbac4eb3746b057 in term 0.
I20260926 01:00:07.356330 10954 leader_election.cc:302] T bf333c7b63a04d6fb39f4811efcfeec0 P 78b75a2199c0452f9bbac4eb3746b057 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47a2277a9fd24c14b23f901d6823409f, 78b75a2199c0452f9bbac4eb3746b057; no voters:
I20260926 01:00:07.356457 11406 raft_consensus.cc:2781] T bf333c7b63a04d6fb39f4811efcfeec0 P 78b75a2199c0452f9bbac4eb3746b057 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:00:07.356513 11406 raft_consensus.cc:482] T bf333c7b63a04d6fb39f4811efcfeec0 P 78b75a2199c0452f9bbac4eb3746b057 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:00:07.356547 11406 raft_consensus.cc:3037] T bf333c7b63a04d6fb39f4811efcfeec0 P 78b75a2199c0452f9bbac4eb3746b057 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:00:07.357275 11406 raft_consensus.cc:504] T bf333c7b63a04d6fb39f4811efcfeec0 P 78b75a2199c0452f9bbac4eb3746b057 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b75a2199c0452f9bbac4eb3746b057" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 45301 } } peers { permanent_uuid: "a42f67fb6fef43c6867b287287170a1f" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39043 } } peers { permanent_uuid: "47a2277a9fd24c14b23f901d6823409f" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46757 } }
I20260926 01:00:07.357431 11406 leader_election.cc:288] T bf333c7b63a04d6fb39f4811efcfeec0 P 78b75a2199c0452f9bbac4eb3746b057 [CANDIDATE]: Term 1 election: Requested vote from peers a42f67fb6fef43c6867b287287170a1f (127.3.75.2:39043), 47a2277a9fd24c14b23f901d6823409f (127.3.75.3:46757)
I20260926 01:00:07.357633 11301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf333c7b63a04d6fb39f4811efcfeec0" candidate_uuid: "78b75a2199c0452f9bbac4eb3746b057" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47a2277a9fd24c14b23f901d6823409f"
I20260926 01:00:07.357633 11161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf333c7b63a04d6fb39f4811efcfeec0" candidate_uuid: "78b75a2199c0452f9bbac4eb3746b057" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a42f67fb6fef43c6867b287287170a1f"
I20260926 01:00:07.357710 11161 raft_consensus.cc:3037] T bf333c7b63a04d6fb39f4811efcfeec0 P a42f67fb6fef43c6867b287287170a1f [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:00:07.357710 11301 raft_consensus.cc:3037] T bf333c7b63a04d6fb39f4811efcfeec0 P 47a2277a9fd24c14b23f901d6823409f [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:00:07.358412 11301 raft_consensus.cc:2454] T bf333c7b63a04d6fb39f4811efcfeec0 P 47a2277a9fd24c14b23f901d6823409f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b75a2199c0452f9bbac4eb3746b057 in term 1.
I20260926 01:00:07.358484 11161 raft_consensus.cc:2454] T bf333c7b63a04d6fb39f4811efcfeec0 P a42f67fb6fef43c6867b287287170a1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b75a2199c0452f9bbac4eb3746b057 in term 1.
I20260926 01:00:07.358594 10954 leader_election.cc:302] T bf333c7b63a04d6fb39f4811efcfeec0 P 78b75a2199c0452f9bbac4eb3746b057 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47a2277a9fd24c14b23f901d6823409f, 78b75a2199c0452f9bbac4eb3746b057; no voters:
I20260926 01:00:07.358671 11406 raft_consensus.cc:2781] T bf333c7b63a04d6fb39f4811efcfeec0 P 78b75a2199c0452f9bbac4eb3746b057 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:00:07.358731 11406 raft_consensus.cc:686] T bf333c7b63a04d6fb39f4811efcfeec0 P 78b75a2199c0452f9bbac4eb3746b057 [term 1 LEADER]: Becoming Leader. State: Replica: 78b75a2199c0452f9bbac4eb3746b057, State: Running, Role: LEADER
I20260926 01:00:07.358783 11406 consensus_queue.cc:237] T bf333c7b63a04d6fb39f4811efcfeec0 P 78b75a2199c0452f9bbac4eb3746b057 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b75a2199c0452f9bbac4eb3746b057" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 45301 } } peers { permanent_uuid: "a42f67fb6fef43c6867b287287170a1f" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39043 } } peers { permanent_uuid: "47a2277a9fd24c14b23f901d6823409f" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46757 } }
I20260926 01:00:07.359146 11382 tablet_replica.cc:439] T bf333c7b63a04d6fb39f4811efcfeec0 P 78b75a2199c0452f9bbac4eb3746b057: TxnStatusTablet state changed. Reason: New leader 78b75a2199c0452f9bbac4eb3746b057. Latest consensus state: current_term: 1 leader_uuid: "78b75a2199c0452f9bbac4eb3746b057" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b75a2199c0452f9bbac4eb3746b057" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 45301 } } peers { permanent_uuid: "a42f67fb6fef43c6867b287287170a1f" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39043 } } peers { permanent_uuid: "47a2277a9fd24c14b23f901d6823409f" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46757 } } }
I20260926 01:00:07.359269 11382 tablet_replica.cc:442] T bf333c7b63a04d6fb39f4811efcfeec0 P 78b75a2199c0452f9bbac4eb3746b057: This TxnStatusTablet replica's current role is: LEADER
I20260926 01:00:07.359483 11418 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 01:00:07.359578 11418 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 01:00:07.359495 10862 catalog_manager.cc:5495] T bf333c7b63a04d6fb39f4811efcfeec0 P 78b75a2199c0452f9bbac4eb3746b057 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78b75a2199c0452f9bbac4eb3746b057 (127.3.75.1). New cstate: current_term: 1 leader_uuid: "78b75a2199c0452f9bbac4eb3746b057" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b75a2199c0452f9bbac4eb3746b057" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 45301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a42f67fb6fef43c6867b287287170a1f" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47a2277a9fd24c14b23f901d6823409f" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46757 } health_report { overall_health: UNKNOWN } } }
I20260926 01:00:07.368517 11371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:07.365850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50324 (local address 127.3.75.1:45301)
0926 01:00:07.365899 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:07.365901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:07.365911 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:00:07.365998 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:07.365999 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:07.366015 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:07.366097 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:00:07.366262 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:07.366818 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:07.367676 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:07.367896 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:07.367986 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:07.368182 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:00:07.368281 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:00:07.368361 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:07.368426 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 01:00:07.368448 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 01:00:07.368526 11413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:07.365786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.1:45301 (local address 127.0.0.1:50324)
0926 01:00:07.365859 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:07.365871 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:07.365920 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:07.366105 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:07.366106 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:07.366108 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 01:00:07.366157 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:07.366159 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:07.366917 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:07.366918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:07.367576 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:07.367578 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:07.367989 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:07.367990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:07.368056 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:07.368075 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 01:00:07.368361 (+ 286us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 01:00:07.368371 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 01:00:07.368412 (+ 41us) client_negotiation.cc:240] Negotiation successful
0926 01:00:07.368457 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 01:00:07.369668 11301 raft_consensus.cc:1261] T bf333c7b63a04d6fb39f4811efcfeec0 P 47a2277a9fd24c14b23f901d6823409f [term 1 FOLLOWER]: Refusing update from remote peer 78b75a2199c0452f9bbac4eb3746b057: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:00:07.369802 11161 raft_consensus.cc:1261] T bf333c7b63a04d6fb39f4811efcfeec0 P a42f67fb6fef43c6867b287287170a1f [term 1 FOLLOWER]: Refusing update from remote peer 78b75a2199c0452f9bbac4eb3746b057: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:00:07.369933 11406 consensus_queue.cc:1035] T bf333c7b63a04d6fb39f4811efcfeec0 P 78b75a2199c0452f9bbac4eb3746b057 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47a2277a9fd24c14b23f901d6823409f" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:00:07.370240 11406 consensus_queue.cc:1035] T bf333c7b63a04d6fb39f4811efcfeec0 P 78b75a2199c0452f9bbac4eb3746b057 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a42f67fb6fef43c6867b287287170a1f" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:00:07.370488 11381 tablet_replica.cc:439] T bf333c7b63a04d6fb39f4811efcfeec0 P 47a2277a9fd24c14b23f901d6823409f: TxnStatusTablet state changed. Reason: New leader 78b75a2199c0452f9bbac4eb3746b057. Latest consensus state: current_term: 1 leader_uuid: "78b75a2199c0452f9bbac4eb3746b057" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b75a2199c0452f9bbac4eb3746b057" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 45301 } } peers { permanent_uuid: "a42f67fb6fef43c6867b287287170a1f" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39043 } } peers { permanent_uuid: "47a2277a9fd24c14b23f901d6823409f" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46757 } } }
I20260926 01:00:07.370580 11381 tablet_replica.cc:442] T bf333c7b63a04d6fb39f4811efcfeec0 P 47a2277a9fd24c14b23f901d6823409f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 01:00:07.370801 11395 tablet_replica.cc:439] T bf333c7b63a04d6fb39f4811efcfeec0 P a42f67fb6fef43c6867b287287170a1f: TxnStatusTablet state changed. Reason: New leader 78b75a2199c0452f9bbac4eb3746b057. Latest consensus state: current_term: 1 leader_uuid: "78b75a2199c0452f9bbac4eb3746b057" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b75a2199c0452f9bbac4eb3746b057" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 45301 } } peers { permanent_uuid: "a42f67fb6fef43c6867b287287170a1f" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39043 } } peers { permanent_uuid: "47a2277a9fd24c14b23f901d6823409f" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46757 } } }
I20260926 01:00:07.370901 11395 tablet_replica.cc:442] T bf333c7b63a04d6fb39f4811efcfeec0 P a42f67fb6fef43c6867b287287170a1f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 01:00:07.371491 11382 tablet_replica.cc:439] T bf333c7b63a04d6fb39f4811efcfeec0 P 78b75a2199c0452f9bbac4eb3746b057: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "78b75a2199c0452f9bbac4eb3746b057" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b75a2199c0452f9bbac4eb3746b057" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 45301 } } peers { permanent_uuid: "a42f67fb6fef43c6867b287287170a1f" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39043 } } peers { permanent_uuid: "47a2277a9fd24c14b23f901d6823409f" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46757 } } }
I20260926 01:00:07.371572 11382 tablet_replica.cc:442] T bf333c7b63a04d6fb39f4811efcfeec0 P 78b75a2199c0452f9bbac4eb3746b057: This TxnStatusTablet replica's current role is: LEADER
I20260926 01:00:07.371506 11406 tablet_replica.cc:439] T bf333c7b63a04d6fb39f4811efcfeec0 P 78b75a2199c0452f9bbac4eb3746b057: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "78b75a2199c0452f9bbac4eb3746b057" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b75a2199c0452f9bbac4eb3746b057" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 45301 } } peers { permanent_uuid: "a42f67fb6fef43c6867b287287170a1f" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39043 } } peers { permanent_uuid: "47a2277a9fd24c14b23f901d6823409f" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46757 } } }
I20260926 01:00:07.372318 11406 tablet_replica.cc:442] T bf333c7b63a04d6fb39f4811efcfeec0 P 78b75a2199c0452f9bbac4eb3746b057: This TxnStatusTablet replica's current role is: LEADER
I20260926 01:00:07.372344 11395 tablet_replica.cc:439] T bf333c7b63a04d6fb39f4811efcfeec0 P a42f67fb6fef43c6867b287287170a1f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "78b75a2199c0452f9bbac4eb3746b057" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b75a2199c0452f9bbac4eb3746b057" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 45301 } } peers { permanent_uuid: "a42f67fb6fef43c6867b287287170a1f" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39043 } } peers { permanent_uuid: "47a2277a9fd24c14b23f901d6823409f" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46757 } } }
I20260926 01:00:07.372422 11395 tablet_replica.cc:442] T bf333c7b63a04d6fb39f4811efcfeec0 P a42f67fb6fef43c6867b287287170a1f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 01:00:07.372565 11381 tablet_replica.cc:439] T bf333c7b63a04d6fb39f4811efcfeec0 P 47a2277a9fd24c14b23f901d6823409f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "78b75a2199c0452f9bbac4eb3746b057" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b75a2199c0452f9bbac4eb3746b057" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 45301 } } peers { permanent_uuid: "a42f67fb6fef43c6867b287287170a1f" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39043 } } peers { permanent_uuid: "47a2277a9fd24c14b23f901d6823409f" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46757 } } }
I20260926 01:00:07.372637 11381 tablet_replica.cc:442] T bf333c7b63a04d6fb39f4811efcfeec0 P 47a2277a9fd24c14b23f901d6823409f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 01:00:07.375861 11371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:07.373504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50332 (local address 127.3.75.1:45301)
0926 01:00:07.373614 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:07.373616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:07.373622 (+ 6us) server_negotiation.cc:407] Connection header received
0926 01:00:07.373640 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:07.373640 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:07.373650 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:07.373699 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:00:07.373840 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:07.374287 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:07.375179 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:07.375387 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:07.375466 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:07.375625 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:00:07.375658 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:00:07.375704 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:07.375773 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 01:00:07.375791 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260926 01:00:07.381024 11371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:07.378331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:47567 (local address 127.3.75.1:45301)
0926 01:00:07.378400 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:07.378402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:07.378545 (+ 143us) server_negotiation.cc:407] Connection header received
0926 01:00:07.378566 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:07.378567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:07.378577 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:07.378627 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:00:07.378784 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:07.379514 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:07.380258 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:07.380554 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:07.380622 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:07.380770 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:00:07.380801 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:00:07.380871 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:07.380921 (+ 50us) server_negotiation.cc:299] Negotiation successful
0926 01:00:07.380938 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 01:00:07.381029 11426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:07.378291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.1:45301 (local address 127.3.75.1:47567)
0926 01:00:07.378452 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:07.378461 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:07.378508 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:07.378640 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:07.378640 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:07.378651 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 01:00:07.378713 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:07.378714 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:07.379591 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:07.379591 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:07.380174 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:07.380176 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:07.380632 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:07.380632 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:07.380677 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:07.380690 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 01:00:07.380883 (+ 193us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 01:00:07.380884 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 01:00:07.380918 (+ 34us) client_negotiation.cc:240] Negotiation successful
0926 01:00:07.380949 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":86,"threads_started":1}
I20260926 01:00:07.383524 11301 raft_consensus.cc:1261] T 7a2c1fb104af4c6c90d949a1a9cb697a P 47a2277a9fd24c14b23f901d6823409f [term 1 FOLLOWER]: Refusing update from remote peer 78b75a2199c0452f9bbac4eb3746b057: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:00:07.383539 11161 raft_consensus.cc:1261] T 7a2c1fb104af4c6c90d949a1a9cb697a P a42f67fb6fef43c6867b287287170a1f [term 1 FOLLOWER]: Refusing update from remote peer 78b75a2199c0452f9bbac4eb3746b057: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:00:07.383752 11382 consensus_queue.cc:1035] T 7a2c1fb104af4c6c90d949a1a9cb697a P 78b75a2199c0452f9bbac4eb3746b057 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a42f67fb6fef43c6867b287287170a1f" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:00:07.383836 11406 consensus_queue.cc:1035] T 7a2c1fb104af4c6c90d949a1a9cb697a P 78b75a2199c0452f9bbac4eb3746b057 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47a2277a9fd24c14b23f901d6823409f" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:00:07.394845 10862 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:00:07.394969 10862 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:00:07.395596 10862 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 246badea3f0f4badadc5380fb63e9019: Sending DeleteTablet for 3 replicas of tablet 7a2c1fb104af4c6c90d949a1a9cb697a
I20260926 01:00:07.395855 11000 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a2c1fb104af4c6c90d949a1a9cb697a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:00:07 UTC) from {username='oryx', principal='oryx/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:50308
I20260926 01:00:07.395892 11281 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a2c1fb104af4c6c90d949a1a9cb697a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:00:07 UTC) from {username='oryx', principal='oryx/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:42006
I20260926 01:00:07.395891 11141 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a2c1fb104af4c6c90d949a1a9cb697a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:00:07 UTC) from {username='oryx', principal='oryx/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:57088
I20260926 01:00:07.395974 11399 tablet_replica.cc:330] T 7a2c1fb104af4c6c90d949a1a9cb697a P 47a2277a9fd24c14b23f901d6823409f: stopping tablet replica
I20260926 01:00:07.395982 11400 tablet_replica.cc:330] T 7a2c1fb104af4c6c90d949a1a9cb697a P 78b75a2199c0452f9bbac4eb3746b057: stopping tablet replica
I20260926 01:00:07.396018 11399 raft_consensus.cc:2229] T 7a2c1fb104af4c6c90d949a1a9cb697a P 47a2277a9fd24c14b23f901d6823409f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:00:07.396034 11400 raft_consensus.cc:2229] T 7a2c1fb104af4c6c90d949a1a9cb697a P 78b75a2199c0452f9bbac4eb3746b057 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:00:07.396046 11399 raft_consensus.cc:2258] T 7a2c1fb104af4c6c90d949a1a9cb697a P 47a2277a9fd24c14b23f901d6823409f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:00:07.396104 11400 raft_consensus.cc:2258] T 7a2c1fb104af4c6c90d949a1a9cb697a P 78b75a2199c0452f9bbac4eb3746b057 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:00:07.396171 11398 tablet_replica.cc:330] T 7a2c1fb104af4c6c90d949a1a9cb697a P a42f67fb6fef43c6867b287287170a1f: stopping tablet replica
I20260926 01:00:07.396216 11398 raft_consensus.cc:2229] T 7a2c1fb104af4c6c90d949a1a9cb697a P a42f67fb6fef43c6867b287287170a1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:00:07.396248 11398 raft_consensus.cc:2258] T 7a2c1fb104af4c6c90d949a1a9cb697a P a42f67fb6fef43c6867b287287170a1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:00:07.396409 11400 ts_tablet_manager.cc:1905] T 7a2c1fb104af4c6c90d949a1a9cb697a P 78b75a2199c0452f9bbac4eb3746b057: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:00:07.396680 11398 ts_tablet_manager.cc:1905] T 7a2c1fb104af4c6c90d949a1a9cb697a P a42f67fb6fef43c6867b287287170a1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:00:07.398142 11400 ts_tablet_manager.cc:1918] T 7a2c1fb104af4c6c90d949a1a9cb697a P 78b75a2199c0452f9bbac4eb3746b057: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 01:00:07.398207 11400 log.cc:1196] T 7a2c1fb104af4c6c90d949a1a9cb697a P 78b75a2199c0452f9bbac4eb3746b057: Deleting WAL directory at /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/ts-0/wal/wals/7a2c1fb104af4c6c90d949a1a9cb697a
I20260926 01:00:07.398208 11398 ts_tablet_manager.cc:1918] T 7a2c1fb104af4c6c90d949a1a9cb697a P a42f67fb6fef43c6867b287287170a1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 01:00:07.398255 11398 log.cc:1196] T 7a2c1fb104af4c6c90d949a1a9cb697a P a42f67fb6fef43c6867b287287170a1f: Deleting WAL directory at /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/ts-1/wal/wals/7a2c1fb104af4c6c90d949a1a9cb697a
I20260926 01:00:07.398447 11400 ts_tablet_manager.cc:1939] T 7a2c1fb104af4c6c90d949a1a9cb697a P 78b75a2199c0452f9bbac4eb3746b057: Deleting consensus metadata
I20260926 01:00:07.398452 11398 ts_tablet_manager.cc:1939] T 7a2c1fb104af4c6c90d949a1a9cb697a P a42f67fb6fef43c6867b287287170a1f: Deleting consensus metadata
I20260926 01:00:07.398785 10851 catalog_manager.cc:4841] TS 78b75a2199c0452f9bbac4eb3746b057 (127.3.75.1:45301): tablet 7a2c1fb104af4c6c90d949a1a9cb697a (table test-table [id=ec2ce6a2677a434b86c8e856a3bd4e8e]) successfully deleted
I20260926 01:00:07.401430 10851 catalog_manager.cc:4841] TS a42f67fb6fef43c6867b287287170a1f (127.3.75.2:39043): tablet 7a2c1fb104af4c6c90d949a1a9cb697a (table test-table [id=ec2ce6a2677a434b86c8e856a3bd4e8e]) successfully deleted
I20260926 01:00:07.401517 11399 ts_tablet_manager.cc:1905] T 7a2c1fb104af4c6c90d949a1a9cb697a P 47a2277a9fd24c14b23f901d6823409f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:00:07.403304 11399 ts_tablet_manager.cc:1918] T 7a2c1fb104af4c6c90d949a1a9cb697a P 47a2277a9fd24c14b23f901d6823409f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 01:00:07.403376 11399 log.cc:1196] T 7a2c1fb104af4c6c90d949a1a9cb697a P 47a2277a9fd24c14b23f901d6823409f: Deleting WAL directory at /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/ts-2/wal/wals/7a2c1fb104af4c6c90d949a1a9cb697a
I20260926 01:00:07.403594 11399 ts_tablet_manager.cc:1939] T 7a2c1fb104af4c6c90d949a1a9cb697a P 47a2277a9fd24c14b23f901d6823409f: Deleting consensus metadata
I20260926 01:00:07.403901 10850 catalog_manager.cc:4841] TS 47a2277a9fd24c14b23f901d6823409f (127.3.75.3:46757): tablet 7a2c1fb104af4c6c90d949a1a9cb697a (table test-table [id=ec2ce6a2677a434b86c8e856a3bd4e8e]) successfully deleted
Sep 26 01:00:07 dist-test-slave-7v10 krb5kdc[10819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 01:00:07 dist-test-slave-7v10 krb5kdc[10819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:00:07 dist-test-slave-7v10 krb5kdc[10819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 01:00:07.422183 10913 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:00:07.415988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59056 (local address 127.3.75.62:33509)
0926 01:00:07.416067 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:07.416070 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:07.416189 (+ 119us) server_negotiation.cc:407] Connection header received
0926 01:00:07.416272 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:07.416273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:07.416283 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:07.416334 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:07.417072 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:07.417562 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:07.418587 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:07.418771 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:07.418821 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:07.422109 (+ 3288us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59056: BlockingRecv error: recv got EOF from 127.0.0.1:59056 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 26 01:00:07 dist-test-slave-7v10 krb5kdc[10819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM
I20260926 01:00:07.430217 10913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:07.423042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59070 (local address 127.3.75.62:33509)
0926 01:00:07.423081 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:07.423083 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:07.423120 (+ 37us) server_negotiation.cc:407] Connection header received
0926 01:00:07.423206 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:07.423207 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:07.423216 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:07.423290 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:07.423711 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:07.424467 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:07.425132 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:07.425421 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:07.425493 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:07.427685 (+ 2192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:07.427686 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:07.427686 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:00:07.427694 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:07.429204 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:07.429520 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:07.429521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:07.429522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:07.429578 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:07.429759 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:07.429760 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:07.429760 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:07.429882 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:07.429973 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:07.430050 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 01:00:07.430119 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 01:00:07.431913 10862 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59070:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:00:07.432055 10862 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:00:07.433509 11281 tablet_service.cc:1467] Processing CreateTablet for tablet e1eed7d995e642ab8f56ca0e9c57d549 (DEFAULT_TABLE table=test-table [id=56e73011c3fd486c8d5107bfa539a67b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:00:07.433509 11000 tablet_service.cc:1467] Processing CreateTablet for tablet e1eed7d995e642ab8f56ca0e9c57d549 (DEFAULT_TABLE table=test-table [id=56e73011c3fd486c8d5107bfa539a67b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:00:07.433550 11141 tablet_service.cc:1467] Processing CreateTablet for tablet e1eed7d995e642ab8f56ca0e9c57d549 (DEFAULT_TABLE table=test-table [id=56e73011c3fd486c8d5107bfa539a67b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:00:07.433642 11281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1eed7d995e642ab8f56ca0e9c57d549. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:00:07.433642 11000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1eed7d995e642ab8f56ca0e9c57d549. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:00:07.433676 11141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1eed7d995e642ab8f56ca0e9c57d549. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:00:07.435046 11376 tablet_bootstrap.cc:492] T e1eed7d995e642ab8f56ca0e9c57d549 P a42f67fb6fef43c6867b287287170a1f: Bootstrap starting.
I20260926 01:00:07.435065 11377 tablet_bootstrap.cc:492] T e1eed7d995e642ab8f56ca0e9c57d549 P 47a2277a9fd24c14b23f901d6823409f: Bootstrap starting.
I20260926 01:00:07.435065 11375 tablet_bootstrap.cc:492] T e1eed7d995e642ab8f56ca0e9c57d549 P 78b75a2199c0452f9bbac4eb3746b057: Bootstrap starting.
I20260926 01:00:07.435369 11376 tablet_bootstrap.cc:654] T e1eed7d995e642ab8f56ca0e9c57d549 P a42f67fb6fef43c6867b287287170a1f: Neither blocks nor log segments found. Creating new log.
I20260926 01:00:07.435459 11375 tablet_bootstrap.cc:654] T e1eed7d995e642ab8f56ca0e9c57d549 P 78b75a2199c0452f9bbac4eb3746b057: Neither blocks nor log segments found. Creating new log.
I20260926 01:00:07.435459 11377 tablet_bootstrap.cc:654] T e1eed7d995e642ab8f56ca0e9c57d549 P 47a2277a9fd24c14b23f901d6823409f: Neither blocks nor log segments found. Creating new log.
I20260926 01:00:07.436228 11375 tablet_bootstrap.cc:492] T e1eed7d995e642ab8f56ca0e9c57d549 P 78b75a2199c0452f9bbac4eb3746b057: No bootstrap required, opened a new log
I20260926 01:00:07.436300 11375 ts_tablet_manager.cc:1397] T e1eed7d995e642ab8f56ca0e9c57d549 P 78b75a2199c0452f9bbac4eb3746b057: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 01:00:07.436421 11377 tablet_bootstrap.cc:492] T e1eed7d995e642ab8f56ca0e9c57d549 P 47a2277a9fd24c14b23f901d6823409f: No bootstrap required, opened a new log
I20260926 01:00:07.436405 11375 raft_consensus.cc:348] T e1eed7d995e642ab8f56ca0e9c57d549 P 78b75a2199c0452f9bbac4eb3746b057 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b75a2199c0452f9bbac4eb3746b057" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 45301 } } peers { permanent_uuid: "a42f67fb6fef43c6867b287287170a1f" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39043 } } peers { permanent_uuid: "47a2277a9fd24c14b23f901d6823409f" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46757 } }
I20260926 01:00:07.436472 11375 raft_consensus.cc:374] T e1eed7d995e642ab8f56ca0e9c57d549 P 78b75a2199c0452f9bbac4eb3746b057 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:00:07.436468 11377 ts_tablet_manager.cc:1397] T e1eed7d995e642ab8f56ca0e9c57d549 P 47a2277a9fd24c14b23f901d6823409f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 01:00:07.436501 11375 raft_consensus.cc:729] T e1eed7d995e642ab8f56ca0e9c57d549 P 78b75a2199c0452f9bbac4eb3746b057 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b75a2199c0452f9bbac4eb3746b057, State: Initialized, Role: FOLLOWER
I20260926 01:00:07.436546 11375 consensus_queue.cc:260] T e1eed7d995e642ab8f56ca0e9c57d549 P 78b75a2199c0452f9bbac4eb3746b057 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b75a2199c0452f9bbac4eb3746b057" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 45301 } } peers { permanent_uuid: "a42f67fb6fef43c6867b287287170a1f" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39043 } } peers { permanent_uuid: "47a2277a9fd24c14b23f901d6823409f" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46757 } }
I20260926 01:00:07.436640 11375 ts_tablet_manager.cc:1428] T e1eed7d995e642ab8f56ca0e9c57d549 P 78b75a2199c0452f9bbac4eb3746b057: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 01:00:07.436616 11377 raft_consensus.cc:348] T e1eed7d995e642ab8f56ca0e9c57d549 P 47a2277a9fd24c14b23f901d6823409f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b75a2199c0452f9bbac4eb3746b057" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 45301 } } peers { permanent_uuid: "a42f67fb6fef43c6867b287287170a1f" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39043 } } peers { permanent_uuid: "47a2277a9fd24c14b23f901d6823409f" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46757 } }
I20260926 01:00:07.436679 11377 raft_consensus.cc:374] T e1eed7d995e642ab8f56ca0e9c57d549 P 47a2277a9fd24c14b23f901d6823409f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:00:07.436704 11377 raft_consensus.cc:729] T e1eed7d995e642ab8f56ca0e9c57d549 P 47a2277a9fd24c14b23f901d6823409f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47a2277a9fd24c14b23f901d6823409f, State: Initialized, Role: FOLLOWER
I20260926 01:00:07.436759 11377 consensus_queue.cc:260] T e1eed7d995e642ab8f56ca0e9c57d549 P 47a2277a9fd24c14b23f901d6823409f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b75a2199c0452f9bbac4eb3746b057" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 45301 } } peers { permanent_uuid: "a42f67fb6fef43c6867b287287170a1f" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39043 } } peers { permanent_uuid: "47a2277a9fd24c14b23f901d6823409f" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46757 } }
I20260926 01:00:07.436837 11377 ts_tablet_manager.cc:1428] T e1eed7d995e642ab8f56ca0e9c57d549 P 47a2277a9fd24c14b23f901d6823409f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:07.437215 11376 tablet_bootstrap.cc:492] T e1eed7d995e642ab8f56ca0e9c57d549 P a42f67fb6fef43c6867b287287170a1f: No bootstrap required, opened a new log
I20260926 01:00:07.437268 11376 ts_tablet_manager.cc:1397] T e1eed7d995e642ab8f56ca0e9c57d549 P a42f67fb6fef43c6867b287287170a1f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 01:00:07.437379 11376 raft_consensus.cc:348] T e1eed7d995e642ab8f56ca0e9c57d549 P a42f67fb6fef43c6867b287287170a1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b75a2199c0452f9bbac4eb3746b057" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 45301 } } peers { permanent_uuid: "a42f67fb6fef43c6867b287287170a1f" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39043 } } peers { permanent_uuid: "47a2277a9fd24c14b23f901d6823409f" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46757 } }
I20260926 01:00:07.437448 11376 raft_consensus.cc:374] T e1eed7d995e642ab8f56ca0e9c57d549 P a42f67fb6fef43c6867b287287170a1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:00:07.437470 11376 raft_consensus.cc:729] T e1eed7d995e642ab8f56ca0e9c57d549 P a42f67fb6fef43c6867b287287170a1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a42f67fb6fef43c6867b287287170a1f, State: Initialized, Role: FOLLOWER
I20260926 01:00:07.437542 11376 consensus_queue.cc:260] T e1eed7d995e642ab8f56ca0e9c57d549 P a42f67fb6fef43c6867b287287170a1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b75a2199c0452f9bbac4eb3746b057" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 45301 } } peers { permanent_uuid: "a42f67fb6fef43c6867b287287170a1f" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39043 } } peers { permanent_uuid: "47a2277a9fd24c14b23f901d6823409f" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46757 } }
I20260926 01:00:07.437636 11376 ts_tablet_manager.cc:1428] T e1eed7d995e642ab8f56ca0e9c57d549 P a42f67fb6fef43c6867b287287170a1f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:07.445802 11395 raft_consensus.cc:482] T e1eed7d995e642ab8f56ca0e9c57d549 P a42f67fb6fef43c6867b287287170a1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:00:07.445887 11395 raft_consensus.cc:504] T e1eed7d995e642ab8f56ca0e9c57d549 P a42f67fb6fef43c6867b287287170a1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b75a2199c0452f9bbac4eb3746b057" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 45301 } } peers { permanent_uuid: "a42f67fb6fef43c6867b287287170a1f" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39043 } } peers { permanent_uuid: "47a2277a9fd24c14b23f901d6823409f" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46757 } }
I20260926 01:00:07.446038 11395 leader_election.cc:288] T e1eed7d995e642ab8f56ca0e9c57d549 P a42f67fb6fef43c6867b287287170a1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78b75a2199c0452f9bbac4eb3746b057 (127.3.75.1:45301), 47a2277a9fd24c14b23f901d6823409f (127.3.75.3:46757)
I20260926 01:00:07.446215 11300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1eed7d995e642ab8f56ca0e9c57d549" candidate_uuid: "a42f67fb6fef43c6867b287287170a1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47a2277a9fd24c14b23f901d6823409f" is_pre_election: true
I20260926 01:00:07.446218 11021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1eed7d995e642ab8f56ca0e9c57d549" candidate_uuid: "a42f67fb6fef43c6867b287287170a1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b75a2199c0452f9bbac4eb3746b057" is_pre_election: true
I20260926 01:00:07.446313 11021 raft_consensus.cc:2454] T e1eed7d995e642ab8f56ca0e9c57d549 P 78b75a2199c0452f9bbac4eb3746b057 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a42f67fb6fef43c6867b287287170a1f in term 0.
I20260926 01:00:07.446313 11300 raft_consensus.cc:2454] T e1eed7d995e642ab8f56ca0e9c57d549 P 47a2277a9fd24c14b23f901d6823409f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a42f67fb6fef43c6867b287287170a1f in term 0.
I20260926 01:00:07.446455 11094 leader_election.cc:302] T e1eed7d995e642ab8f56ca0e9c57d549 P a42f67fb6fef43c6867b287287170a1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47a2277a9fd24c14b23f901d6823409f, a42f67fb6fef43c6867b287287170a1f; no voters:
I20260926 01:00:07.446566 11395 raft_consensus.cc:2781] T e1eed7d995e642ab8f56ca0e9c57d549 P a42f67fb6fef43c6867b287287170a1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:00:07.446607 11395 raft_consensus.cc:482] T e1eed7d995e642ab8f56ca0e9c57d549 P a42f67fb6fef43c6867b287287170a1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:00:07.446633 11395 raft_consensus.cc:3037] T e1eed7d995e642ab8f56ca0e9c57d549 P a42f67fb6fef43c6867b287287170a1f [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:00:07.447431 11395 raft_consensus.cc:504] T e1eed7d995e642ab8f56ca0e9c57d549 P a42f67fb6fef43c6867b287287170a1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b75a2199c0452f9bbac4eb3746b057" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 45301 } } peers { permanent_uuid: "a42f67fb6fef43c6867b287287170a1f" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39043 } } peers { permanent_uuid: "47a2277a9fd24c14b23f901d6823409f" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46757 } }
I20260926 01:00:07.447576 11395 leader_election.cc:288] T e1eed7d995e642ab8f56ca0e9c57d549 P a42f67fb6fef43c6867b287287170a1f [CANDIDATE]: Term 1 election: Requested vote from peers 78b75a2199c0452f9bbac4eb3746b057 (127.3.75.1:45301), 47a2277a9fd24c14b23f901d6823409f (127.3.75.3:46757)
I20260926 01:00:07.447712 11021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1eed7d995e642ab8f56ca0e9c57d549" candidate_uuid: "a42f67fb6fef43c6867b287287170a1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b75a2199c0452f9bbac4eb3746b057"
I20260926 01:00:07.447816 11021 raft_consensus.cc:3037] T e1eed7d995e642ab8f56ca0e9c57d549 P 78b75a2199c0452f9bbac4eb3746b057 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:00:07.447813 11300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1eed7d995e642ab8f56ca0e9c57d549" candidate_uuid: "a42f67fb6fef43c6867b287287170a1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47a2277a9fd24c14b23f901d6823409f"
I20260926 01:00:07.447878 11300 raft_consensus.cc:3037] T e1eed7d995e642ab8f56ca0e9c57d549 P 47a2277a9fd24c14b23f901d6823409f [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:00:07.448563 11021 raft_consensus.cc:2454] T e1eed7d995e642ab8f56ca0e9c57d549 P 78b75a2199c0452f9bbac4eb3746b057 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a42f67fb6fef43c6867b287287170a1f in term 1.
I20260926 01:00:07.448737 11095 leader_election.cc:302] T e1eed7d995e642ab8f56ca0e9c57d549 P a42f67fb6fef43c6867b287287170a1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b75a2199c0452f9bbac4eb3746b057, a42f67fb6fef43c6867b287287170a1f; no voters:
I20260926 01:00:07.448840 11395 raft_consensus.cc:2781] T e1eed7d995e642ab8f56ca0e9c57d549 P a42f67fb6fef43c6867b287287170a1f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:00:07.448894 11300 raft_consensus.cc:2454] T e1eed7d995e642ab8f56ca0e9c57d549 P 47a2277a9fd24c14b23f901d6823409f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a42f67fb6fef43c6867b287287170a1f in term 1.
I20260926 01:00:07.448961 11395 raft_consensus.cc:686] T e1eed7d995e642ab8f56ca0e9c57d549 P a42f67fb6fef43c6867b287287170a1f [term 1 LEADER]: Becoming Leader. State: Replica: a42f67fb6fef43c6867b287287170a1f, State: Running, Role: LEADER
I20260926 01:00:07.449018 11395 consensus_queue.cc:237] T e1eed7d995e642ab8f56ca0e9c57d549 P a42f67fb6fef43c6867b287287170a1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b75a2199c0452f9bbac4eb3746b057" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 45301 } } peers { permanent_uuid: "a42f67fb6fef43c6867b287287170a1f" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39043 } } peers { permanent_uuid: "47a2277a9fd24c14b23f901d6823409f" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46757 } }
I20260926 01:00:07.449625 10864 catalog_manager.cc:5495] T e1eed7d995e642ab8f56ca0e9c57d549 P a42f67fb6fef43c6867b287287170a1f reported cstate change: term changed from 0 to 1, leader changed from <none> to a42f67fb6fef43c6867b287287170a1f (127.3.75.2). New cstate: current_term: 1 leader_uuid: "a42f67fb6fef43c6867b287287170a1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b75a2199c0452f9bbac4eb3746b057" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 45301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a42f67fb6fef43c6867b287287170a1f" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39043 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47a2277a9fd24c14b23f901d6823409f" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46757 } health_report { overall_health: UNKNOWN } } }
I20260926 01:00:07.455096 11373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:07.452471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57116 (local address 127.3.75.2:39043)
0926 01:00:07.452520 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:07.452523 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:07.452570 (+ 47us) server_negotiation.cc:407] Connection header received
0926 01:00:07.452639 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:07.452640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:07.452654 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:07.452715 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:00:07.452809 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:07.453515 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:07.454257 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:07.454542 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:07.454623 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:07.454758 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:00:07.454794 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:00:07.454852 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:07.455002 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 01:00:07.455025 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 01:00:07.455754 11021 raft_consensus.cc:1261] T e1eed7d995e642ab8f56ca0e9c57d549 P 78b75a2199c0452f9bbac4eb3746b057 [term 1 FOLLOWER]: Refusing update from remote peer a42f67fb6fef43c6867b287287170a1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:00:07.455768 11300 raft_consensus.cc:1261] T e1eed7d995e642ab8f56ca0e9c57d549 P 47a2277a9fd24c14b23f901d6823409f [term 1 FOLLOWER]: Refusing update from remote peer a42f67fb6fef43c6867b287287170a1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:00:07.455945 11445 consensus_queue.cc:1035] T e1eed7d995e642ab8f56ca0e9c57d549 P a42f67fb6fef43c6867b287287170a1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b75a2199c0452f9bbac4eb3746b057" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 45301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:00:07.456017 11395 consensus_queue.cc:1035] T e1eed7d995e642ab8f56ca0e9c57d549 P a42f67fb6fef43c6867b287287170a1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "47a2277a9fd24c14b23f901d6823409f" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:00:07.458570 10862 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:00:07.458658 10862 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:00:07.459395 10862 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 246badea3f0f4badadc5380fb63e9019: Sending DeleteTablet for 3 replicas of tablet e1eed7d995e642ab8f56ca0e9c57d549
I20260926 01:00:07.459635 11281 tablet_service.cc:1514] Processing DeleteTablet for tablet e1eed7d995e642ab8f56ca0e9c57d549 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:00:07 UTC) from {username='oryx', principal='oryx/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:42006
I20260926 01:00:07.459681 11000 tablet_service.cc:1514] Processing DeleteTablet for tablet e1eed7d995e642ab8f56ca0e9c57d549 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:00:07 UTC) from {username='oryx', principal='oryx/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:50308
I20260926 01:00:07.459721 11399 tablet_replica.cc:330] T e1eed7d995e642ab8f56ca0e9c57d549 P 47a2277a9fd24c14b23f901d6823409f: stopping tablet replica
I20260926 01:00:07.459765 11399 raft_consensus.cc:2229] T e1eed7d995e642ab8f56ca0e9c57d549 P 47a2277a9fd24c14b23f901d6823409f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:00:07.459779 11400 tablet_replica.cc:330] T e1eed7d995e642ab8f56ca0e9c57d549 P 78b75a2199c0452f9bbac4eb3746b057: stopping tablet replica
I20260926 01:00:07.459803 11399 raft_consensus.cc:2258] T e1eed7d995e642ab8f56ca0e9c57d549 P 47a2277a9fd24c14b23f901d6823409f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:00:07.459833 11400 raft_consensus.cc:2229] T e1eed7d995e642ab8f56ca0e9c57d549 P 78b75a2199c0452f9bbac4eb3746b057 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:00:07.459883 11400 raft_consensus.cc:2258] T e1eed7d995e642ab8f56ca0e9c57d549 P 78b75a2199c0452f9bbac4eb3746b057 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:00:07.460012 11141 tablet_service.cc:1514] Processing DeleteTablet for tablet e1eed7d995e642ab8f56ca0e9c57d549 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:00:07 UTC) from {username='oryx', principal='oryx/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:57088
I20260926 01:00:07.460127 11398 tablet_replica.cc:330] T e1eed7d995e642ab8f56ca0e9c57d549 P a42f67fb6fef43c6867b287287170a1f: stopping tablet replica
I20260926 01:00:07.460192 11398 raft_consensus.cc:2229] T e1eed7d995e642ab8f56ca0e9c57d549 P a42f67fb6fef43c6867b287287170a1f [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:00:07.460249 11398 raft_consensus.cc:2258] T e1eed7d995e642ab8f56ca0e9c57d549 P a42f67fb6fef43c6867b287287170a1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:00:07.460433 11399 ts_tablet_manager.cc:1905] T e1eed7d995e642ab8f56ca0e9c57d549 P 47a2277a9fd24c14b23f901d6823409f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:00:07.460502 11400 ts_tablet_manager.cc:1905] T e1eed7d995e642ab8f56ca0e9c57d549 P 78b75a2199c0452f9bbac4eb3746b057: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:00:07.461802 11399 ts_tablet_manager.cc:1918] T e1eed7d995e642ab8f56ca0e9c57d549 P 47a2277a9fd24c14b23f901d6823409f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:00:07.461862 11399 log.cc:1196] T e1eed7d995e642ab8f56ca0e9c57d549 P 47a2277a9fd24c14b23f901d6823409f: Deleting WAL directory at /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/ts-2/wal/wals/e1eed7d995e642ab8f56ca0e9c57d549
I20260926 01:00:07.461798 10862 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59070:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:00:07.461930 10862 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:00:07.462011 11400 ts_tablet_manager.cc:1918] T e1eed7d995e642ab8f56ca0e9c57d549 P 78b75a2199c0452f9bbac4eb3746b057: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:00:07.462083 11400 log.cc:1196] T e1eed7d995e642ab8f56ca0e9c57d549 P 78b75a2199c0452f9bbac4eb3746b057: Deleting WAL directory at /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/ts-0/wal/wals/e1eed7d995e642ab8f56ca0e9c57d549
I20260926 01:00:07.462095 11399 ts_tablet_manager.cc:1939] T e1eed7d995e642ab8f56ca0e9c57d549 P 47a2277a9fd24c14b23f901d6823409f: Deleting consensus metadata
I20260926 01:00:07.462276 11400 ts_tablet_manager.cc:1939] T e1eed7d995e642ab8f56ca0e9c57d549 P 78b75a2199c0452f9bbac4eb3746b057: Deleting consensus metadata
I20260926 01:00:07.462404 10850 catalog_manager.cc:4841] TS 47a2277a9fd24c14b23f901d6823409f (127.3.75.3:46757): tablet e1eed7d995e642ab8f56ca0e9c57d549 (table test-table [id=56e73011c3fd486c8d5107bfa539a67b]) successfully deleted
I20260926 01:00:07.462774 10851 catalog_manager.cc:4841] TS 78b75a2199c0452f9bbac4eb3746b057 (127.3.75.1:45301): tablet e1eed7d995e642ab8f56ca0e9c57d549 (table test-table [id=56e73011c3fd486c8d5107bfa539a67b]) successfully deleted
I20260926 01:00:07.463150 11398 ts_tablet_manager.cc:1905] T e1eed7d995e642ab8f56ca0e9c57d549 P a42f67fb6fef43c6867b287287170a1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:00:07.463476 11281 tablet_service.cc:1467] Processing CreateTablet for tablet 3170a93668724baba9e7507d88a3850a (DEFAULT_TABLE table=test-table [id=3158691556ef4ec0ae86b387b1fbae2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:00:07.463577 11281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3170a93668724baba9e7507d88a3850a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:00:07.464635 11398 ts_tablet_manager.cc:1918] T e1eed7d995e642ab8f56ca0e9c57d549 P a42f67fb6fef43c6867b287287170a1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:00:07.464686 11398 log.cc:1196] T e1eed7d995e642ab8f56ca0e9c57d549 P a42f67fb6fef43c6867b287287170a1f: Deleting WAL directory at /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/ts-1/wal/wals/e1eed7d995e642ab8f56ca0e9c57d549
I20260926 01:00:07.464831 11377 tablet_bootstrap.cc:492] T 3170a93668724baba9e7507d88a3850a P 47a2277a9fd24c14b23f901d6823409f: Bootstrap starting.
I20260926 01:00:07.464912 11398 ts_tablet_manager.cc:1939] T e1eed7d995e642ab8f56ca0e9c57d549 P a42f67fb6fef43c6867b287287170a1f: Deleting consensus metadata
I20260926 01:00:07.465237 11377 tablet_bootstrap.cc:654] T 3170a93668724baba9e7507d88a3850a P 47a2277a9fd24c14b23f901d6823409f: Neither blocks nor log segments found. Creating new log.
I20260926 01:00:07.465942 11377 tablet_bootstrap.cc:492] T 3170a93668724baba9e7507d88a3850a P 47a2277a9fd24c14b23f901d6823409f: No bootstrap required, opened a new log
I20260926 01:00:07.465993 11377 ts_tablet_manager.cc:1397] T 3170a93668724baba9e7507d88a3850a P 47a2277a9fd24c14b23f901d6823409f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 01:00:07.466144 11377 raft_consensus.cc:348] T 3170a93668724baba9e7507d88a3850a P 47a2277a9fd24c14b23f901d6823409f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b75a2199c0452f9bbac4eb3746b057" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 45301 } } peers { permanent_uuid: "a42f67fb6fef43c6867b287287170a1f" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39043 } } peers { permanent_uuid: "47a2277a9fd24c14b23f901d6823409f" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46757 } }
I20260926 01:00:07.466230 11377 raft_consensus.cc:374] T 3170a93668724baba9e7507d88a3850a P 47a2277a9fd24c14b23f901d6823409f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:00:07.466252 11377 raft_consensus.cc:729] T 3170a93668724baba9e7507d88a3850a P 47a2277a9fd24c14b23f901d6823409f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47a2277a9fd24c14b23f901d6823409f, State: Initialized, Role: FOLLOWER
I20260926 01:00:07.466293 11377 consensus_queue.cc:260] T 3170a93668724baba9e7507d88a3850a P 47a2277a9fd24c14b23f901d6823409f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b75a2199c0452f9bbac4eb3746b057" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 45301 } } peers { permanent_uuid: "a42f67fb6fef43c6867b287287170a1f" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39043 } } peers { permanent_uuid: "47a2277a9fd24c14b23f901d6823409f" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46757 } }
I20260926 01:00:07.466384 11377 ts_tablet_manager.cc:1428] T 3170a93668724baba9e7507d88a3850a P 47a2277a9fd24c14b23f901d6823409f: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 01:00:07.466655 10851 catalog_manager.cc:4841] TS a42f67fb6fef43c6867b287287170a1f (127.3.75.2:39043): tablet e1eed7d995e642ab8f56ca0e9c57d549 (table test-table [id=56e73011c3fd486c8d5107bfa539a67b]) successfully deleted
I20260926 01:00:07.466835 11000 tablet_service.cc:1467] Processing CreateTablet for tablet 3170a93668724baba9e7507d88a3850a (DEFAULT_TABLE table=test-table [id=3158691556ef4ec0ae86b387b1fbae2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:00:07.466972 11000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3170a93668724baba9e7507d88a3850a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:00:07.468161 11375 tablet_bootstrap.cc:492] T 3170a93668724baba9e7507d88a3850a P 78b75a2199c0452f9bbac4eb3746b057: Bootstrap starting.
I20260926 01:00:07.468194 11141 tablet_service.cc:1467] Processing CreateTablet for tablet 3170a93668724baba9e7507d88a3850a (DEFAULT_TABLE table=test-table [id=3158691556ef4ec0ae86b387b1fbae2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:00:07.468289 11141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3170a93668724baba9e7507d88a3850a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:00:07.468495 11375 tablet_bootstrap.cc:654] T 3170a93668724baba9e7507d88a3850a P 78b75a2199c0452f9bbac4eb3746b057: Neither blocks nor log segments found. Creating new log.
I20260926 01:00:07.469090 11375 tablet_bootstrap.cc:492] T 3170a93668724baba9e7507d88a3850a P 78b75a2199c0452f9bbac4eb3746b057: No bootstrap required, opened a new log
I20260926 01:00:07.469156 11375 ts_tablet_manager.cc:1397] T 3170a93668724baba9e7507d88a3850a P 78b75a2199c0452f9bbac4eb3746b057: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 01:00:07.469313 11375 raft_consensus.cc:348] T 3170a93668724baba9e7507d88a3850a P 78b75a2199c0452f9bbac4eb3746b057 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b75a2199c0452f9bbac4eb3746b057" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 45301 } } peers { permanent_uuid: "a42f67fb6fef43c6867b287287170a1f" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39043 } } peers { permanent_uuid: "47a2277a9fd24c14b23f901d6823409f" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46757 } }
I20260926 01:00:07.469390 11375 raft_consensus.cc:374] T 3170a93668724baba9e7507d88a3850a P 78b75a2199c0452f9bbac4eb3746b057 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:00:07.469419 11375 raft_consensus.cc:729] T 3170a93668724baba9e7507d88a3850a P 78b75a2199c0452f9bbac4eb3746b057 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b75a2199c0452f9bbac4eb3746b057, State: Initialized, Role: FOLLOWER
I20260926 01:00:07.469447 11376 tablet_bootstrap.cc:492] T 3170a93668724baba9e7507d88a3850a P a42f67fb6fef43c6867b287287170a1f: Bootstrap starting.
I20260926 01:00:07.469463 11375 consensus_queue.cc:260] T 3170a93668724baba9e7507d88a3850a P 78b75a2199c0452f9bbac4eb3746b057 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b75a2199c0452f9bbac4eb3746b057" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 45301 } } peers { permanent_uuid: "a42f67fb6fef43c6867b287287170a1f" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39043 } } peers { permanent_uuid: "47a2277a9fd24c14b23f901d6823409f" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46757 } }
I20260926 01:00:07.469555 11375 ts_tablet_manager.cc:1428] T 3170a93668724baba9e7507d88a3850a P 78b75a2199c0452f9bbac4eb3746b057: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 01:00:07.469794 11376 tablet_bootstrap.cc:654] T 3170a93668724baba9e7507d88a3850a P a42f67fb6fef43c6867b287287170a1f: Neither blocks nor log segments found. Creating new log.
I20260926 01:00:07.470543 11376 tablet_bootstrap.cc:492] T 3170a93668724baba9e7507d88a3850a P a42f67fb6fef43c6867b287287170a1f: No bootstrap required, opened a new log
I20260926 01:00:07.470616 11376 ts_tablet_manager.cc:1397] T 3170a93668724baba9e7507d88a3850a P a42f67fb6fef43c6867b287287170a1f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 01:00:07.470722 11376 raft_consensus.cc:348] T 3170a93668724baba9e7507d88a3850a P a42f67fb6fef43c6867b287287170a1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b75a2199c0452f9bbac4eb3746b057" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 45301 } } peers { permanent_uuid: "a42f67fb6fef43c6867b287287170a1f" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39043 } } peers { permanent_uuid: "47a2277a9fd24c14b23f901d6823409f" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46757 } }
I20260926 01:00:07.470822 11376 raft_consensus.cc:374] T 3170a93668724baba9e7507d88a3850a P a42f67fb6fef43c6867b287287170a1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:00:07.470855 11376 raft_consensus.cc:729] T 3170a93668724baba9e7507d88a3850a P a42f67fb6fef43c6867b287287170a1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a42f67fb6fef43c6867b287287170a1f, State: Initialized, Role: FOLLOWER
I20260926 01:00:07.470927 11376 consensus_queue.cc:260] T 3170a93668724baba9e7507d88a3850a P a42f67fb6fef43c6867b287287170a1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b75a2199c0452f9bbac4eb3746b057" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 45301 } } peers { permanent_uuid: "a42f67fb6fef43c6867b287287170a1f" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39043 } } peers { permanent_uuid: "47a2277a9fd24c14b23f901d6823409f" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46757 } }
I20260926 01:00:07.471014 11376 ts_tablet_manager.cc:1428] T 3170a93668724baba9e7507d88a3850a P a42f67fb6fef43c6867b287287170a1f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 01:00:07.639603 11445 raft_consensus.cc:482] T 3170a93668724baba9e7507d88a3850a P a42f67fb6fef43c6867b287287170a1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:00:07.639760 11445 raft_consensus.cc:504] T 3170a93668724baba9e7507d88a3850a P a42f67fb6fef43c6867b287287170a1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b75a2199c0452f9bbac4eb3746b057" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 45301 } } peers { permanent_uuid: "a42f67fb6fef43c6867b287287170a1f" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39043 } } peers { permanent_uuid: "47a2277a9fd24c14b23f901d6823409f" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46757 } }
I20260926 01:00:07.639935 11445 leader_election.cc:288] T 3170a93668724baba9e7507d88a3850a P a42f67fb6fef43c6867b287287170a1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78b75a2199c0452f9bbac4eb3746b057 (127.3.75.1:45301), 47a2277a9fd24c14b23f901d6823409f (127.3.75.3:46757)
I20260926 01:00:07.640231 11300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3170a93668724baba9e7507d88a3850a" candidate_uuid: "a42f67fb6fef43c6867b287287170a1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47a2277a9fd24c14b23f901d6823409f" is_pre_election: true
I20260926 01:00:07.640349 11300 raft_consensus.cc:2454] T 3170a93668724baba9e7507d88a3850a P 47a2277a9fd24c14b23f901d6823409f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a42f67fb6fef43c6867b287287170a1f in term 0.
I20260926 01:00:07.640456 11021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3170a93668724baba9e7507d88a3850a" candidate_uuid: "a42f67fb6fef43c6867b287287170a1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b75a2199c0452f9bbac4eb3746b057" is_pre_election: true
I20260926 01:00:07.640551 11021 raft_consensus.cc:2454] T 3170a93668724baba9e7507d88a3850a P 78b75a2199c0452f9bbac4eb3746b057 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a42f67fb6fef43c6867b287287170a1f in term 0.
I20260926 01:00:07.640511 11094 leader_election.cc:302] T 3170a93668724baba9e7507d88a3850a P a42f67fb6fef43c6867b287287170a1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47a2277a9fd24c14b23f901d6823409f, a42f67fb6fef43c6867b287287170a1f; no voters:
I20260926 01:00:07.640667 11445 raft_consensus.cc:2781] T 3170a93668724baba9e7507d88a3850a P a42f67fb6fef43c6867b287287170a1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:00:07.640709 11445 raft_consensus.cc:482] T 3170a93668724baba9e7507d88a3850a P a42f67fb6fef43c6867b287287170a1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:00:07.640734 11445 raft_consensus.cc:3037] T 3170a93668724baba9e7507d88a3850a P a42f67fb6fef43c6867b287287170a1f [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:00:07.641856 11445 raft_consensus.cc:504] T 3170a93668724baba9e7507d88a3850a P a42f67fb6fef43c6867b287287170a1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b75a2199c0452f9bbac4eb3746b057" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 45301 } } peers { permanent_uuid: "a42f67fb6fef43c6867b287287170a1f" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39043 } } peers { permanent_uuid: "47a2277a9fd24c14b23f901d6823409f" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46757 } }
I20260926 01:00:07.642009 11445 leader_election.cc:288] T 3170a93668724baba9e7507d88a3850a P a42f67fb6fef43c6867b287287170a1f [CANDIDATE]: Term 1 election: Requested vote from peers 78b75a2199c0452f9bbac4eb3746b057 (127.3.75.1:45301), 47a2277a9fd24c14b23f901d6823409f (127.3.75.3:46757)
I20260926 01:00:07.642189 11300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3170a93668724baba9e7507d88a3850a" candidate_uuid: "a42f67fb6fef43c6867b287287170a1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47a2277a9fd24c14b23f901d6823409f"
I20260926 01:00:07.642272 11300 raft_consensus.cc:3037] T 3170a93668724baba9e7507d88a3850a P 47a2277a9fd24c14b23f901d6823409f [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:00:07.642192 11021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3170a93668724baba9e7507d88a3850a" candidate_uuid: "a42f67fb6fef43c6867b287287170a1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b75a2199c0452f9bbac4eb3746b057"
I20260926 01:00:07.642318 11021 raft_consensus.cc:3037] T 3170a93668724baba9e7507d88a3850a P 78b75a2199c0452f9bbac4eb3746b057 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:00:07.643180 11300 raft_consensus.cc:2454] T 3170a93668724baba9e7507d88a3850a P 47a2277a9fd24c14b23f901d6823409f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a42f67fb6fef43c6867b287287170a1f in term 1.
I20260926 01:00:07.643193 11021 raft_consensus.cc:2454] T 3170a93668724baba9e7507d88a3850a P 78b75a2199c0452f9bbac4eb3746b057 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a42f67fb6fef43c6867b287287170a1f in term 1.
I20260926 01:00:07.643388 11095 leader_election.cc:302] T 3170a93668724baba9e7507d88a3850a P a42f67fb6fef43c6867b287287170a1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b75a2199c0452f9bbac4eb3746b057, a42f67fb6fef43c6867b287287170a1f; no voters:
I20260926 01:00:07.643522 11445 raft_consensus.cc:2781] T 3170a93668724baba9e7507d88a3850a P a42f67fb6fef43c6867b287287170a1f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:00:07.643576 11445 raft_consensus.cc:686] T 3170a93668724baba9e7507d88a3850a P a42f67fb6fef43c6867b287287170a1f [term 1 LEADER]: Becoming Leader. State: Replica: a42f67fb6fef43c6867b287287170a1f, State: Running, Role: LEADER
I20260926 01:00:07.643633 11445 consensus_queue.cc:237] T 3170a93668724baba9e7507d88a3850a P a42f67fb6fef43c6867b287287170a1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b75a2199c0452f9bbac4eb3746b057" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 45301 } } peers { permanent_uuid: "a42f67fb6fef43c6867b287287170a1f" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39043 } } peers { permanent_uuid: "47a2277a9fd24c14b23f901d6823409f" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46757 } }
I20260926 01:00:07.644229 10862 catalog_manager.cc:5495] T 3170a93668724baba9e7507d88a3850a P a42f67fb6fef43c6867b287287170a1f reported cstate change: term changed from 0 to 1, leader changed from <none> to a42f67fb6fef43c6867b287287170a1f (127.3.75.2). New cstate: current_term: 1 leader_uuid: "a42f67fb6fef43c6867b287287170a1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b75a2199c0452f9bbac4eb3746b057" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 45301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a42f67fb6fef43c6867b287287170a1f" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39043 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47a2277a9fd24c14b23f901d6823409f" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46757 } health_report { overall_health: UNKNOWN } } }
I20260926 01:00:07.661603 11371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:07.659292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:49157 (local address 127.3.75.1:45301)
0926 01:00:07.659350 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:07.659353 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:07.659460 (+ 107us) server_negotiation.cc:407] Connection header received
0926 01:00:07.659497 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:07.659498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:07.659506 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:07.659551 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:00:07.659636 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:07.660094 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:07.661028 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:07.661199 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:07.661250 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:07.661417 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:00:07.661441 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:00:07.661479 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:07.661536 (+ 57us) server_negotiation.cc:299] Negotiation successful
0926 01:00:07.661550 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 01:00:07.661636 11456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:07.659233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.1:45301 (local address 127.3.75.2:49157)
0926 01:00:07.659385 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:07.659396 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:07.659458 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:07.659553 (+ 95us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:07.659554 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:07.659555 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 01:00:07.659602 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:07.659603 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:07.660155 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:07.660155 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:07.660944 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:07.660947 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:07.661282 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:07.661282 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:07.661329 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:07.661343 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 01:00:07.661499 (+ 156us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 01:00:07.661501 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 01:00:07.661535 (+ 34us) client_negotiation.cc:240] Negotiation successful
0926 01:00:07.661574 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":77,"threads_started":1}
I20260926 01:00:07.663574 11300 raft_consensus.cc:1261] T 3170a93668724baba9e7507d88a3850a P 47a2277a9fd24c14b23f901d6823409f [term 1 FOLLOWER]: Refusing update from remote peer a42f67fb6fef43c6867b287287170a1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:00:07.663614 11021 raft_consensus.cc:1261] T 3170a93668724baba9e7507d88a3850a P 78b75a2199c0452f9bbac4eb3746b057 [term 1 FOLLOWER]: Refusing update from remote peer a42f67fb6fef43c6867b287287170a1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:00:07.663818 11395 consensus_queue.cc:1035] T 3170a93668724baba9e7507d88a3850a P a42f67fb6fef43c6867b287287170a1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b75a2199c0452f9bbac4eb3746b057" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 45301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:00:07.663942 11445 consensus_queue.cc:1035] T 3170a93668724baba9e7507d88a3850a P a42f67fb6fef43c6867b287287170a1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "47a2277a9fd24c14b23f901d6823409f" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:00:07.670648 11373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:07.668440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:40045 (local address 127.3.75.2:39043)
0926 01:00:07.668486 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:07.668488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:07.668547 (+ 59us) server_negotiation.cc:407] Connection header received
0926 01:00:07.668610 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:07.668610 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:07.668622 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:07.668671 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:00:07.668771 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:07.669241 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:07.669980 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:07.670199 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:07.670279 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:07.670423 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:00:07.670459 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:00:07.670506 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:07.670562 (+ 56us) server_negotiation.cc:299] Negotiation successful
0926 01:00:07.670579 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 01:00:07.670688 11426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:07.668398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.2:39043 (local address 127.3.75.1:40045)
0926 01:00:07.668481 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:07.668495 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:07.668559 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:07.668672 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:07.668673 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:07.668674 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 01:00:07.668715 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:07.668716 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:07.669330 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:07.669332 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:07.669888 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:07.669889 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:07.670278 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:07.670279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:07.670334 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:07.670339 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 01:00:07.670505 (+ 166us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 01:00:07.670507 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 01:00:07.670558 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 01:00:07.670612 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260926 01:00:07.676365 10864 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:00:07.676448 10864 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:00:07.677021 10864 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 246badea3f0f4badadc5380fb63e9019: Sending DeleteTablet for 3 replicas of tablet 3170a93668724baba9e7507d88a3850a
I20260926 01:00:07.677253 11000 tablet_service.cc:1514] Processing DeleteTablet for tablet 3170a93668724baba9e7507d88a3850a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:00:07 UTC) from {username='oryx', principal='oryx/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:50308
I20260926 01:00:07.677307 11141 tablet_service.cc:1514] Processing DeleteTablet for tablet 3170a93668724baba9e7507d88a3850a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:00:07 UTC) from {username='oryx', principal='oryx/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:57088
I20260926 01:00:07.677349 11400 tablet_replica.cc:330] T 3170a93668724baba9e7507d88a3850a P 78b75a2199c0452f9bbac4eb3746b057: stopping tablet replica
I20260926 01:00:07.677320 11281 tablet_service.cc:1514] Processing DeleteTablet for tablet 3170a93668724baba9e7507d88a3850a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:00:07 UTC) from {username='oryx', principal='oryx/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:42006
I20260926 01:00:07.677397 11400 raft_consensus.cc:2229] T 3170a93668724baba9e7507d88a3850a P 78b75a2199c0452f9bbac4eb3746b057 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:00:07.677415 11398 tablet_replica.cc:330] T 3170a93668724baba9e7507d88a3850a P a42f67fb6fef43c6867b287287170a1f: stopping tablet replica
I20260926 01:00:07.677430 11400 raft_consensus.cc:2258] T 3170a93668724baba9e7507d88a3850a P 78b75a2199c0452f9bbac4eb3746b057 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:00:07.677438 11399 tablet_replica.cc:330] T 3170a93668724baba9e7507d88a3850a P 47a2277a9fd24c14b23f901d6823409f: stopping tablet replica
I20260926 01:00:07.677469 11398 raft_consensus.cc:2229] T 3170a93668724baba9e7507d88a3850a P a42f67fb6fef43c6867b287287170a1f [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:00:07.677487 11399 raft_consensus.cc:2229] T 3170a93668724baba9e7507d88a3850a P 47a2277a9fd24c14b23f901d6823409f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:00:07.677520 11398 raft_consensus.cc:2258] T 3170a93668724baba9e7507d88a3850a P a42f67fb6fef43c6867b287287170a1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:00:07.677521 11399 raft_consensus.cc:2258] T 3170a93668724baba9e7507d88a3850a P 47a2277a9fd24c14b23f901d6823409f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:00:07.677860 11399 ts_tablet_manager.cc:1905] T 3170a93668724baba9e7507d88a3850a P 47a2277a9fd24c14b23f901d6823409f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:00:07.677922 11400 ts_tablet_manager.cc:1905] T 3170a93668724baba9e7507d88a3850a P 78b75a2199c0452f9bbac4eb3746b057: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:00:07.677961 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 10936
I20260926 01:00:07.679513 11399 ts_tablet_manager.cc:1918] T 3170a93668724baba9e7507d88a3850a P 47a2277a9fd24c14b23f901d6823409f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 01:00:07.679592 11399 log.cc:1196] T 3170a93668724baba9e7507d88a3850a P 47a2277a9fd24c14b23f901d6823409f: Deleting WAL directory at /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/ts-2/wal/wals/3170a93668724baba9e7507d88a3850a
I20260926 01:00:07.679873 11399 ts_tablet_manager.cc:1939] T 3170a93668724baba9e7507d88a3850a P 47a2277a9fd24c14b23f901d6823409f: Deleting consensus metadata
I20260926 01:00:07.680157 11398 ts_tablet_manager.cc:1905] T 3170a93668724baba9e7507d88a3850a P a42f67fb6fef43c6867b287287170a1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:00:07.680158 10850 catalog_manager.cc:4841] TS 47a2277a9fd24c14b23f901d6823409f (127.3.75.3:46757): tablet 3170a93668724baba9e7507d88a3850a (table test-table [id=3158691556ef4ec0ae86b387b1fbae2e]) successfully deleted
I20260926 01:00:07.681365 11398 ts_tablet_manager.cc:1918] T 3170a93668724baba9e7507d88a3850a P a42f67fb6fef43c6867b287287170a1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 01:00:07.681416 11398 log.cc:1196] T 3170a93668724baba9e7507d88a3850a P a42f67fb6fef43c6867b287287170a1f: Deleting WAL directory at /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384372073861-3372-0/minicluster-data/ts-1/wal/wals/3170a93668724baba9e7507d88a3850a
I20260926 01:00:07.681628 11398 ts_tablet_manager.cc:1939] T 3170a93668724baba9e7507d88a3850a P a42f67fb6fef43c6867b287287170a1f: Deleting consensus metadata
I20260926 01:00:07.681963 10851 catalog_manager.cc:4841] TS a42f67fb6fef43c6867b287287170a1f (127.3.75.2:39043): tablet 3170a93668724baba9e7507d88a3850a (table test-table [id=3158691556ef4ec0ae86b387b1fbae2e]) successfully deleted
W20260926 01:00:07.685607 10851 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.75.1:45301 (error 108)
I20260926 01:00:07.685792 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 11076
I20260926 01:00:07.687142 11461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:07.685991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.1:45301 (local address 127.0.0.1:50336)
0926 01:00:07.687043 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:07.687092 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.1:45301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1032,"thread_start_us":43,"threads_started":1}
W20260926 01:00:07.687232 10851 catalog_manager.cc:4584] TS 78b75a2199c0452f9bbac4eb3746b057 (127.3.75.1:45301): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3170a93668724baba9e7507d88a3850a: Network error: Client connection negotiation failed: client connection to 127.3.75.1:45301: connect: Connection refused (error 111)
I20260926 01:00:07.691526 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 11216
I20260926 01:00:07.696292 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 10832
2026-09-26T01:00:07Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2749 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790384372073861-3372-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:00:07 dist-test-slave-7v10 krb5kdc[11464](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:00:07 dist-test-slave-7v10 krb5kdc[11464](info): set up 2 sockets
Sep 26 01:00:07 dist-test-slave-7v10 krb5kdc[11464](info): commencing operation
krb5kdc: starting...
W20260926 01:00:08.719633 3372 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Sep 26 01:00:08 dist-test-slave-7v10 krb5kdc[11464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:00:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:00:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.75.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab.
Entry for principal kudu/127.3.75.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab.
I20260926 01:00:08.837433 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790384372073861-3372-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790384372073861-3372-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790384372073861-3372-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:38415
--webserver_interface=127.3.75.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:44877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.75.62:38415
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:00:08.917392 11477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:00:08.917600 11477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:00:08.917622 11477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:00:08.918689 11477 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:00:08.918722 11477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:00:08.918736 11477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:00:08.918748 11477 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:00:08.918761 11477 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:00:08.920142 11477 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:44877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790384372073861-3372-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790384372073861-3372-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.75.62:38415
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:38415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790384372073861-3372-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:00:08.920387 11477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:00:08.920562 11477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:00:08.925477 11482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:00:08.925503 11483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:00:08.925629 11485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:00:08.925863 11477 server_base.cc:1034] running on GCE node
I20260926 01:00:08.926070 11477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:00:08.926388 11477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:00:08.927546 11477 hybrid_clock.cc:648] HybridClock initialized: now 1790384408927523 us; error 37 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-26T01:00:08Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1238 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:00:08 dist-test-slave-7v10 krb5kdc[11490](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:00:08 dist-test-slave-7v10 krb5kdc[11490](info): set up 2 sockets
Sep 26 01:00:08 dist-test-slave-7v10 krb5kdc[11490](info): commencing operation
krb5kdc: starting...
W20260926 01:00:09.998720 3372 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Sep 26 01:00:10 dist-test-slave-7v10 krb5kdc[11490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:00:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:00:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.75.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab.
Entry for principal kudu/127.3.75.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab.
I20260926 01:00:10.119607 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:45681
--webserver_interface=127.3.75.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:43391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.75.62:45681
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:00:10.198792 11503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:00:10.199077 11503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:00:10.199119 11503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:00:10.200227 11503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:00:10.200260 11503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:00:10.200273 11503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:00:10.200304 11503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:00:10.200316 11503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:00:10.201701 11503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:43391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.75.62:45681
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:45681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:00:10.201893 11503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:00:10.202050 11503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:00:10.206918 11508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:00:10.206943 11509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:00:10.207093 11511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:00:10.207041 11503 server_base.cc:1034] running on GCE node
I20260926 01:00:10.207409 11503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:00:10.207649 11503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:00:10.208869 11503 hybrid_clock.cc:648] HybridClock initialized: now 1790384410208855 us; error 30 us; skew 500 ppm
Sep 26 01:00:10 dist-test-slave-7v10 krb5kdc[11490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384410, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:00:10.223078 11503 init.cc:377] Logged in from keytab as kudu/127.3.75.62@KRBTEST.COM (short username kudu)
I20260926 01:00:10.223632 11503 webserver.cc:460] Webserver started at http://127.3.75.62:36741/ using document root <none> and password file <none>
I20260926 01:00:10.223887 11503 fs_manager.cc:359] Metadata directory not provided
I20260926 01:00:10.223963 11503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:00:10.224123 11503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:00:10.225212 11503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/master-0/data/instance:
uuid: "f3224f7f89c54556bb862f8108f9bfd3"
format_stamp: "Formatted at 2026-09-26 01:00:10 on dist-test-slave-7v10"
server_key: "9a575d90c4e7aac39d1f9072bb63b7c9"
server_key_iv: "01d63a882ef495afc9b26dcd031cbc51"
server_key_version: "encryptionkey@0"
I20260926 01:00:10.225580 11503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/master-0/wal/instance:
uuid: "f3224f7f89c54556bb862f8108f9bfd3"
format_stamp: "Formatted at 2026-09-26 01:00:10 on dist-test-slave-7v10"
server_key: "9a575d90c4e7aac39d1f9072bb63b7c9"
server_key_iv: "01d63a882ef495afc9b26dcd031cbc51"
server_key_version: "encryptionkey@0"
I20260926 01:00:10.226999 11503 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260926 01:00:10.227756 11518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:10.227965 11503 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:00:10.228072 11503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/master-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/master-0/wal
uuid: "f3224f7f89c54556bb862f8108f9bfd3"
format_stamp: "Formatted at 2026-09-26 01:00:10 on dist-test-slave-7v10"
server_key: "9a575d90c4e7aac39d1f9072bb63b7c9"
server_key_iv: "01d63a882ef495afc9b26dcd031cbc51"
server_key_version: "encryptionkey@0"
I20260926 01:00:10.228158 11503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:00:10.243067 11503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:00:10.245146 11503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:00:10.245350 11503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:00:10.249673 11503 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.62:45681
I20260926 01:00:10.249707 11570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.62:45681 every 8 connection(s)
I20260926 01:00:10.250109 11503 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
I20260926 01:00:10.251108 11571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:00:10.253502 11571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3224f7f89c54556bb862f8108f9bfd3: Bootstrap starting.
I20260926 01:00:10.254052 11571 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3224f7f89c54556bb862f8108f9bfd3: Neither blocks nor log segments found. Creating new log.
I20260926 01:00:10.254311 11571 log.cc:824] T 00000000000000000000000000000000 P f3224f7f89c54556bb862f8108f9bfd3: Log is configured to *not* fsync() on all Append() calls
I20260926 01:00:10.254843 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 11503
I20260926 01:00:10.255034 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/master-0/wal/instance
I20260926 01:00:10.255249 3372 external_mini_cluster.cc:1428] Setting key b07d77baeecd80e9b735ba5891499de3
I20260926 01:00:10.255298 11571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3224f7f89c54556bb862f8108f9bfd3: No bootstrap required, opened a new log
I20260926 01:00:10.257246 11571 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3224f7f89c54556bb862f8108f9bfd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3224f7f89c54556bb862f8108f9bfd3" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 45681 } }
I20260926 01:00:10.257444 11571 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3224f7f89c54556bb862f8108f9bfd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:00:10.257488 11571 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3224f7f89c54556bb862f8108f9bfd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3224f7f89c54556bb862f8108f9bfd3, State: Initialized, Role: FOLLOWER
I20260926 01:00:10.257637 11571 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3224f7f89c54556bb862f8108f9bfd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3224f7f89c54556bb862f8108f9bfd3" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 45681 } }
I20260926 01:00:10.257737 11571 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3224f7f89c54556bb862f8108f9bfd3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:00:10.257781 11571 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3224f7f89c54556bb862f8108f9bfd3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:00:10.257835 11571 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3224f7f89c54556bb862f8108f9bfd3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:00:10.258524 11571 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3224f7f89c54556bb862f8108f9bfd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3224f7f89c54556bb862f8108f9bfd3" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 45681 } }
I20260926 01:00:10.258658 11571 leader_election.cc:302] T 00000000000000000000000000000000 P f3224f7f89c54556bb862f8108f9bfd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3224f7f89c54556bb862f8108f9bfd3; no voters:
I20260926 01:00:10.258924 11571 leader_election.cc:288] T 00000000000000000000000000000000 P f3224f7f89c54556bb862f8108f9bfd3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:00:10.258960 11576 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3224f7f89c54556bb862f8108f9bfd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:00:10.259128 11576 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3224f7f89c54556bb862f8108f9bfd3 [term 1 LEADER]: Becoming Leader. State: Replica: f3224f7f89c54556bb862f8108f9bfd3, State: Running, Role: LEADER
I20260926 01:00:10.259204 11571 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3224f7f89c54556bb862f8108f9bfd3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:00:10.259276 11576 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3224f7f89c54556bb862f8108f9bfd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3224f7f89c54556bb862f8108f9bfd3" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 45681 } }
I20260926 01:00:10.259661 11576 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3224f7f89c54556bb862f8108f9bfd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3224f7f89c54556bb862f8108f9bfd3. Latest consensus state: current_term: 1 leader_uuid: "f3224f7f89c54556bb862f8108f9bfd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3224f7f89c54556bb862f8108f9bfd3" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 45681 } } }
I20260926 01:00:10.259696 11577 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3224f7f89c54556bb862f8108f9bfd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3224f7f89c54556bb862f8108f9bfd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3224f7f89c54556bb862f8108f9bfd3" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 45681 } } }
I20260926 01:00:10.259804 11577 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3224f7f89c54556bb862f8108f9bfd3 [sys.catalog]: This master's current role is: LEADER
I20260926 01:00:10.259991 11576 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3224f7f89c54556bb862f8108f9bfd3 [sys.catalog]: This master's current role is: LEADER
I20260926 01:00:10.260164 11585 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:00:10.260918 11585 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:00:10.262768 11585 catalog_manager.cc:1347] Generated new cluster ID: 1d7837b5e3f04e0e91a4a4f58478383f
I20260926 01:00:10.262821 11585 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 01:00:10 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 01:00:10.268345 11575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:10.255822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50384 (local address 127.3.75.62:45681)
0926 01:00:10.256014 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:10.256022 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:10.256050 (+ 28us) server_negotiation.cc:407] Connection header received
0926 01:00:10.257209 (+ 1159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:10.257216 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:10.257274 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:10.257397 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:10.258230 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:10.259944 (+ 1714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:10.261070 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:10.261646 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:10.261764 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:10.264517 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:10.264521 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:10.264521 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:00:10.264533 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:10.266560 (+ 2027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:10.266986 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:10.266989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:10.266989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:10.267109 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:10.267325 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:10.267326 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:10.267326 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:10.267643 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:10.267843 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:10.267942 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 01:00:10.268125 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":103,"threads_started":1}
I20260926 01:00:10.277343 11585 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:00:10.277876 11585 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:00:10.280419 11585 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3224f7f89c54556bb862f8108f9bfd3: Generated new TSK 0
I20260926 01:00:10.280627 11585 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.75.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab.
Entry for principal kudu/127.3.75.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab.
I20260926 01:00:10.311902 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.1:0
--local_ip_for_outbound_sockets=127.3.75.1
--webserver_interface=127.3.75.1
--webserver_port=0
--tserver_master_addrs=127.3.75.62:45681
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab
--principal=kudu/127.3.75.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:43391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:00:10.392880 11597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:00:10.393071 11597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:00:10.393093 11597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:00:10.394193 11597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:00:10.394228 11597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:00:10.394254 11597 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.1
I20260926 01:00:10.395684 11597 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:43391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab
--principal=kudu/127.3.75.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.75.1
--webserver_port=0
--tserver_master_addrs=127.3.75.62:45681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.1
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:00:10.395931 11597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:00:10.396111 11597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:00:10.401088 11605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:00:10.401075 11602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:00:10.401067 11603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:00:10.401304 11597 server_base.cc:1034] running on GCE node
I20260926 01:00:10.401577 11597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:00:10.401839 11597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:00:10.403008 11597 hybrid_clock.cc:648] HybridClock initialized: now 1790384410402988 us; error 37 us; skew 500 ppm
Sep 26 01:00:10 dist-test-slave-7v10 krb5kdc[11490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384410, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:00:10.405390 11597 init.cc:377] Logged in from keytab as kudu/127.3.75.1@KRBTEST.COM (short username kudu)
I20260926 01:00:10.405820 11597 webserver.cc:460] Webserver started at http://127.3.75.1:42507/ using document root <none> and password file <none>
I20260926 01:00:10.406049 11597 fs_manager.cc:359] Metadata directory not provided
I20260926 01:00:10.406121 11597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:00:10.406270 11597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:00:10.407361 11597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-0/data/instance:
uuid: "c852054a20f3429b92b160cadcfd4bbb"
format_stamp: "Formatted at 2026-09-26 01:00:10 on dist-test-slave-7v10"
server_key: "338e42623a8d9c33d68dc877adbc229d"
server_key_iv: "588cebd306e14b095dcaf208907732f0"
server_key_version: "encryptionkey@0"
I20260926 01:00:10.407728 11597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-0/wal/instance:
uuid: "c852054a20f3429b92b160cadcfd4bbb"
format_stamp: "Formatted at 2026-09-26 01:00:10 on dist-test-slave-7v10"
server_key: "338e42623a8d9c33d68dc877adbc229d"
server_key_iv: "588cebd306e14b095dcaf208907732f0"
server_key_version: "encryptionkey@0"
I20260926 01:00:10.409015 11597 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:00:10.409698 11612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:10.409864 11597 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:00:10.409964 11597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-0/wal
uuid: "c852054a20f3429b92b160cadcfd4bbb"
format_stamp: "Formatted at 2026-09-26 01:00:10 on dist-test-slave-7v10"
server_key: "338e42623a8d9c33d68dc877adbc229d"
server_key_iv: "588cebd306e14b095dcaf208907732f0"
server_key_version: "encryptionkey@0"
I20260926 01:00:10.410043 11597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:00:10.426764 11597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:00:10.428721 11597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:00:10.428920 11597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:00:10.429209 11597 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:00:10.429574 11597 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:00:10.429637 11597 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:10.429682 11597 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:00:10.429708 11597 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:10.435911 11597 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.1:33045
I20260926 01:00:10.435947 11725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.1:33045 every 8 connection(s)
I20260926 01:00:10.436295 11597 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
I20260926 01:00:10.436514 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 11597
I20260926 01:00:10.436625 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-0/wal/instance
I20260926 01:00:10.436851 3372 external_mini_cluster.cc:1428] Setting key 19a4684810a7b619fca7e25d879608b7
Sep 26 01:00:10 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384410, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.1@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.75.2@KRBTEST.COM; defaulting to no policy
I20260926 01:00:10.446581 11575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:10.437104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:35361 (local address 127.3.75.62:45681)
0926 01:00:10.437184 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:10.437186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:10.437451 (+ 265us) server_negotiation.cc:407] Connection header received
0926 01:00:10.438004 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:10.438005 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:10.438020 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:10.438105 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:10.439140 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:10.439641 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:10.440671 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:10.440884 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:10.441002 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:10.443507 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:10.443508 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:10.443508 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:00:10.443516 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:10.444990 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:10.445461 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:10.445462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:10.445463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:10.445526 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:10.445817 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:10.445818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:10.445819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:10.445972 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:10.446043 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:10.446354 (+ 311us) server_negotiation.cc:299] Negotiation successful
0926 01:00:10.446457 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 01:00:10.446750 11728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:10.437114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:45681 (local address 127.3.75.1:35361)
0926 01:00:10.437311 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:10.437357 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:10.437898 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:10.438154 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:10.438157 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:10.438714 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:10.439025 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:10.439033 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:10.439751 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:10.439752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:10.440581 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:10.440584 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:10.441017 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:10.441019 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:10.441186 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:10.441699 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:10.441707 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:10.443381 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:00:10.445116 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:10.445124 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:10.445126 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:10.445358 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:10.445621 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:10.445622 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:10.445622 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:10.445738 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:10.446034 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:00:10.446049 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:00:10.446186 (+ 137us) client_negotiation.cc:769] Sending connection context
0926 01:00:10.446302 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 01:00:10.446514 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":100,"threads_started":1}
I20260926 01:00:10.447414 11726 heartbeater.cc:347] Connected to a master server at 127.3.75.62:45681
I20260926 01:00:10.447503 11726 heartbeater.cc:464] Registering TS with master...
I20260926 01:00:10.447695 11726 heartbeater.cc:511] Master 127.3.75.62:45681 requested a full tablet report, sending...
I20260926 01:00:10.448330 11535 ts_manager.cc:194] Registered new tserver with Master: c852054a20f3429b92b160cadcfd4bbb (127.3.75.1:33045)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.2@KRBTEST.COM" created.
I20260926 01:00:10.449174 11535 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.75.1@KRBTEST.COM'} at 127.3.75.1:35361
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab.
Entry for principal kudu/127.3.75.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab.
I20260926 01:00:10.463896 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.2:0
--local_ip_for_outbound_sockets=127.3.75.2
--webserver_interface=127.3.75.2
--webserver_port=0
--tserver_master_addrs=127.3.75.62:45681
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab
--principal=kudu/127.3.75.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:43391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:00:10.544391 11731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:00:10.544589 11731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:00:10.544612 11731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:00:10.545697 11731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:00:10.545730 11731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:00:10.545754 11731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.2
I20260926 01:00:10.547044 11731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:43391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab
--principal=kudu/127.3.75.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.75.2
--webserver_port=0
--tserver_master_addrs=127.3.75.62:45681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.2
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:00:10.547240 11731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:00:10.547399 11731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:00:10.552253 11739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:00:10.552268 11736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:00:10.552268 11737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:00:10.552444 11731 server_base.cc:1034] running on GCE node
I20260926 01:00:10.552686 11731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:00:10.552922 11731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:00:10.554095 11731 hybrid_clock.cc:648] HybridClock initialized: now 1790384410554071 us; error 37 us; skew 500 ppm
Sep 26 01:00:10 dist-test-slave-7v10 krb5kdc[11490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384410, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:00:10.556828 11731 init.cc:377] Logged in from keytab as kudu/127.3.75.2@KRBTEST.COM (short username kudu)
I20260926 01:00:10.557267 11731 webserver.cc:460] Webserver started at http://127.3.75.2:38915/ using document root <none> and password file <none>
I20260926 01:00:10.557503 11731 fs_manager.cc:359] Metadata directory not provided
I20260926 01:00:10.557574 11731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:00:10.557709 11731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:00:10.558744 11731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-1/data/instance:
uuid: "b72d4ac549cb474b9f939575bc51f61b"
format_stamp: "Formatted at 2026-09-26 01:00:10 on dist-test-slave-7v10"
server_key: "e3e2d0d75e2e12f45c24823fbd3ca3da"
server_key_iv: "9f5939b46fd0b6a3ca24dcec5f3c9bca"
server_key_version: "encryptionkey@0"
I20260926 01:00:10.559155 11731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-1/wal/instance:
uuid: "b72d4ac549cb474b9f939575bc51f61b"
format_stamp: "Formatted at 2026-09-26 01:00:10 on dist-test-slave-7v10"
server_key: "e3e2d0d75e2e12f45c24823fbd3ca3da"
server_key_iv: "9f5939b46fd0b6a3ca24dcec5f3c9bca"
server_key_version: "encryptionkey@0"
I20260926 01:00:10.560555 11731 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:00:10.561272 11746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:10.561436 11731 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:00:10.561537 11731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-1/wal
uuid: "b72d4ac549cb474b9f939575bc51f61b"
format_stamp: "Formatted at 2026-09-26 01:00:10 on dist-test-slave-7v10"
server_key: "e3e2d0d75e2e12f45c24823fbd3ca3da"
server_key_iv: "9f5939b46fd0b6a3ca24dcec5f3c9bca"
server_key_version: "encryptionkey@0"
I20260926 01:00:10.561617 11731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:00:10.570348 11731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:00:10.572103 11731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:00:10.572286 11731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:00:10.572502 11731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:00:10.572831 11731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:00:10.572888 11731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:10.572921 11731 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:00:10.572947 11731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:10.578465 11731 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.2:40981
I20260926 01:00:10.578557 11859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.2:40981 every 8 connection(s)
I20260926 01:00:10.578860 11731 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
Sep 26 01:00:10 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384410, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.2@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 01:00:10.587893 11575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:10.579616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:55047 (local address 127.3.75.62:45681)
0926 01:00:10.579728 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:10.579730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:10.579887 (+ 157us) server_negotiation.cc:407] Connection header received
0926 01:00:10.580402 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:10.580403 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:10.580414 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:10.580474 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:10.581404 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:10.581911 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:10.582988 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:10.583200 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:10.583295 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:10.585267 (+ 1972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:10.585268 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:10.585269 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:00:10.585276 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:10.586456 (+ 1180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:10.586869 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:10.586919 (+ 50us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:10.586920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:10.586978 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:10.587243 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:10.587244 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:10.587244 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:10.587362 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:10.587448 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:10.587715 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 01:00:10.587804 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 01:00:10.588050 11862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:10.579545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:45681 (local address 127.3.75.2:55047)
0926 01:00:10.579757 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:10.579800 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:10.580292 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:10.580538 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:10.580542 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:10.581108 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:10.581295 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:10.581299 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:10.582032 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:10.582033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:10.582819 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:10.582821 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:10.583306 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:10.583307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:10.583423 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:10.583836 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:10.583844 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:10.585128 (+ 1284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:00:10.586567 (+ 1439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:10.586574 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:10.586575 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:10.586787 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:10.587054 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:10.587055 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:10.587055 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:10.587165 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:10.587437 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:00:10.587444 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:00:10.587578 (+ 134us) client_negotiation.cc:769] Sending connection context
0926 01:00:10.587686 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 01:00:10.587831 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":84,"threads_started":1}
I20260926 01:00:10.588415 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 11731
I20260926 01:00:10.588542 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-1/wal/instance
I20260926 01:00:10.588570 11860 heartbeater.cc:347] Connected to a master server at 127.3.75.62:45681
I20260926 01:00:10.588660 11860 heartbeater.cc:464] Registering TS with master...
I20260926 01:00:10.588774 3372 external_mini_cluster.cc:1428] Setting key c9c8fafd740438de760ea815971689f0
I20260926 01:00:10.588826 11860 heartbeater.cc:511] Master 127.3.75.62:45681 requested a full tablet report, sending...
I20260926 01:00:10.589226 11534 ts_manager.cc:194] Registered new tserver with Master: b72d4ac549cb474b9f939575bc51f61b (127.3.75.2:40981)
I20260926 01:00:10.589649 11534 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.75.2@KRBTEST.COM'} at 127.3.75.2:55047
WARNING: no policy specified for kudu/127.3.75.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab.
Entry for principal kudu/127.3.75.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab.
I20260926 01:00:10.613736 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.3:0
--local_ip_for_outbound_sockets=127.3.75.3
--webserver_interface=127.3.75.3
--webserver_port=0
--tserver_master_addrs=127.3.75.62:45681
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab
--principal=kudu/127.3.75.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:43391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:00:10.694453 11865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:00:10.694658 11865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:00:10.694681 11865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:00:10.695928 11865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:00:10.695972 11865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:00:10.696053 11865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.3
I20260926 01:00:10.697469 11865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:43391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab
--principal=kudu/127.3.75.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.75.3
--webserver_port=0
--tserver_master_addrs=127.3.75.62:45681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.3
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:00:10.697718 11865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:00:10.697940 11865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:00:10.702915 11871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:00:10.702955 11870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:00:10.702927 11873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:00:10.703066 11865 server_base.cc:1034] running on GCE node
I20260926 01:00:10.703365 11865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:00:10.703605 11865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:00:10.704792 11865 hybrid_clock.cc:648] HybridClock initialized: now 1790384410704772 us; error 38 us; skew 500 ppm
Sep 26 01:00:10 dist-test-slave-7v10 krb5kdc[11490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384410, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:00:10.707285 11865 init.cc:377] Logged in from keytab as kudu/127.3.75.3@KRBTEST.COM (short username kudu)
I20260926 01:00:10.707702 11865 webserver.cc:460] Webserver started at http://127.3.75.3:35345/ using document root <none> and password file <none>
I20260926 01:00:10.707935 11865 fs_manager.cc:359] Metadata directory not provided
I20260926 01:00:10.708020 11865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:00:10.708161 11865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:00:10.709173 11865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-2/data/instance:
uuid: "492ff811e66140739750d238cb27accc"
format_stamp: "Formatted at 2026-09-26 01:00:10 on dist-test-slave-7v10"
server_key: "c4e22a504efd354df51d55a0d5f1568a"
server_key_iv: "5a84304576a7f778f048994626d39e32"
server_key_version: "encryptionkey@0"
I20260926 01:00:10.709548 11865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-2/wal/instance:
uuid: "492ff811e66140739750d238cb27accc"
format_stamp: "Formatted at 2026-09-26 01:00:10 on dist-test-slave-7v10"
server_key: "c4e22a504efd354df51d55a0d5f1568a"
server_key_iv: "5a84304576a7f778f048994626d39e32"
server_key_version: "encryptionkey@0"
I20260926 01:00:10.710844 11865 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 01:00:10.711669 11880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:10.711922 11865 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:00:10.712011 11865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-2/wal
uuid: "492ff811e66140739750d238cb27accc"
format_stamp: "Formatted at 2026-09-26 01:00:10 on dist-test-slave-7v10"
server_key: "c4e22a504efd354df51d55a0d5f1568a"
server_key_iv: "5a84304576a7f778f048994626d39e32"
server_key_version: "encryptionkey@0"
I20260926 01:00:10.712064 11865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:00:10.727252 11865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:00:10.729120 11865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:00:10.729236 11865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:00:10.729409 11865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:00:10.729732 11865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:00:10.729766 11865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:10.729789 11865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:00:10.729846 11865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:10.735615 11865 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.3:44383
I20260926 01:00:10.735703 11993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.3:44383 every 8 connection(s)
I20260926 01:00:10.736073 11865 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
I20260926 01:00:10.738538 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 11865
I20260926 01:00:10.738663 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-2/wal/instance
I20260926 01:00:10.738866 3372 external_mini_cluster.cc:1428] Setting key eec8007a64d71f67df377f8affdb7ca0
Sep 26 01:00:10 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384410, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.3@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 01:00:10.745360 11575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:10.736858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.3:44343 (local address 127.3.75.62:45681)
0926 01:00:10.736906 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:10.736909 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:10.737131 (+ 222us) server_negotiation.cc:407] Connection header received
0926 01:00:10.737583 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:10.737584 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:10.737595 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:10.737648 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:10.738571 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:10.739297 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:10.740464 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:10.740644 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:10.740712 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:10.742850 (+ 2138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:10.742851 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:10.742852 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:00:10.742858 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:10.744177 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:10.744555 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:10.744556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:10.744556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:10.744606 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:10.744828 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:10.744828 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:10.744829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:10.744929 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:10.744980 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:10.745226 (+ 246us) server_negotiation.cc:299] Negotiation successful
0926 01:00:10.745295 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 01:00:10.745522 11996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:10.736848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:45681 (local address 127.3.75.3:44343)
0926 01:00:10.737008 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:10.737042 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:10.737495 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:10.737708 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:10.737712 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:10.738254 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:10.738463 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:10.738468 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:10.739374 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:10.739375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:10.740380 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:10.740383 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:10.740735 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:10.740736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:10.740846 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:10.741402 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:10.741412 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:10.742763 (+ 1351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:00:10.744261 (+ 1498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:10.744272 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:10.744274 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:10.744484 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:10.744666 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:10.744667 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:10.744667 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:10.744769 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:10.744986 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:00:10.744991 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:00:10.745102 (+ 111us) client_negotiation.cc:769] Sending connection context
0926 01:00:10.745183 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 01:00:10.745303 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":83,"threads_started":1}
I20260926 01:00:10.745982 11994 heartbeater.cc:347] Connected to a master server at 127.3.75.62:45681
I20260926 01:00:10.746089 11994 heartbeater.cc:464] Registering TS with master...
I20260926 01:00:10.746259 11994 heartbeater.cc:511] Master 127.3.75.62:45681 requested a full tablet report, sending...
I20260926 01:00:10.746639 11534 ts_manager.cc:194] Registered new tserver with Master: 492ff811e66140739750d238cb27accc (127.3.75.3:44383)
I20260926 01:00:10.747109 11534 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.75.3@KRBTEST.COM'} at 127.3.75.3:44343
I20260926 01:00:10.750526 3372 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.3.75.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.75.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.75.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/krb5kdc/oryx_127.3.75.1.keytab.
Entry for principal oryx/127.3.75.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/krb5kdc/oryx_127.3.75.1.keytab.
WARNING: no policy specified for oryx/127.3.75.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.75.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.75.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/krb5kdc/oryx_127.3.75.2.keytab.
Entry for principal oryx/127.3.75.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/krb5kdc/oryx_127.3.75.2.keytab.
WARNING: no policy specified for oryx/127.3.75.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.75.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.75.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/krb5kdc/oryx_127.3.75.3.keytab.
Entry for principal oryx/127.3.75.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/krb5kdc/oryx_127.3.75.3.keytab.
I20260926 01:00:10.821031 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 11597
I20260926 01:00:10.826462 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 11731
I20260926 01:00:10.830641 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 11865
I20260926 01:00:10.834668 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 11503
I20260926 01:00:10.839497 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:45681
--webserver_interface=127.3.75.62
--webserver_port=36741
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:43391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.75.62:45681
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:00:10.919190 12003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:00:10.919426 12003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:00:10.919466 12003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:00:10.920608 12003 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:00:10.920683 12003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:00:10.920715 12003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:00:10.920749 12003 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:00:10.920780 12003 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:00:10.922341 12003 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:43391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.75.62:45681
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:45681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.75.62
--webserver_port=36741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:00:10.922623 12003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:00:10.922894 12003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:00:10.928483 12011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:00:10.928469 12008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:00:10.928517 12009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:00:10.928773 12003 server_base.cc:1034] running on GCE node
I20260926 01:00:10.929024 12003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:00:10.929354 12003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:00:10.930509 12003 hybrid_clock.cc:648] HybridClock initialized: now 1790384410930489 us; error 44 us; skew 500 ppm
Sep 26 01:00:10 dist-test-slave-7v10 krb5kdc[11490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384410, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:00:10.933111 12003 init.cc:377] Logged in from keytab as kudu/127.3.75.62@KRBTEST.COM (short username kudu)
I20260926 01:00:10.933706 12003 webserver.cc:460] Webserver started at http://127.3.75.62:36741/ using document root <none> and password file <none>
I20260926 01:00:10.933971 12003 fs_manager.cc:359] Metadata directory not provided
I20260926 01:00:10.934064 12003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:00:10.935527 12003 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:00:10.936192 12018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:10.936358 12003 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:00:10.936471 12003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/master-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/master-0/wal
uuid: "f3224f7f89c54556bb862f8108f9bfd3"
format_stamp: "Formatted at 2026-09-26 01:00:10 on dist-test-slave-7v10"
server_key: "9a575d90c4e7aac39d1f9072bb63b7c9"
server_key_iv: "01d63a882ef495afc9b26dcd031cbc51"
server_key_version: "encryptionkey@0"
I20260926 01:00:10.936830 12003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:00:10.944159 12003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:00:10.945889 12003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:00:10.946046 12003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:00:10.950165 12003 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.62:45681
I20260926 01:00:10.950193 12070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.62:45681 every 8 connection(s)
I20260926 01:00:10.950558 12003 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
I20260926 01:00:10.951982 12071 sys_catalog.cc:263] Verifying existing consensus state
I20260926 01:00:10.952610 12071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3224f7f89c54556bb862f8108f9bfd3: Bootstrap starting.
I20260926 01:00:10.953882 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 12003
I20260926 01:00:10.954173 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.1:33045
--local_ip_for_outbound_sockets=127.3.75.1
--tserver_master_addrs=127.3.75.62:45681
--webserver_port=42507
--webserver_interface=127.3.75.1
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab
--principal=kudu/127.3.75.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:43391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.75.1
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/krb5kdc/oryx_127.3.75.1.keytab with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 01:00:10.960935 12071 log.cc:824] T 00000000000000000000000000000000 P f3224f7f89c54556bb862f8108f9bfd3: Log is configured to *not* fsync() on all Append() calls
I20260926 01:00:10.962532 12071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3224f7f89c54556bb862f8108f9bfd3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 01:00:10.962827 12071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3224f7f89c54556bb862f8108f9bfd3: Bootstrap complete.
I20260926 01:00:10.965049 12071 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3224f7f89c54556bb862f8108f9bfd3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3224f7f89c54556bb862f8108f9bfd3" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 45681 } }
I20260926 01:00:10.965302 12071 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3224f7f89c54556bb862f8108f9bfd3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3224f7f89c54556bb862f8108f9bfd3, State: Initialized, Role: FOLLOWER
I20260926 01:00:10.965487 12071 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3224f7f89c54556bb862f8108f9bfd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3224f7f89c54556bb862f8108f9bfd3" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 45681 } }
I20260926 01:00:10.965570 12071 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3224f7f89c54556bb862f8108f9bfd3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:00:10.965622 12071 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3224f7f89c54556bb862f8108f9bfd3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:00:10.965668 12071 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3224f7f89c54556bb862f8108f9bfd3 [term 1 FOLLOWER]: Advancing to term 2
I20260926 01:00:10.966614 12071 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3224f7f89c54556bb862f8108f9bfd3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3224f7f89c54556bb862f8108f9bfd3" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 45681 } }
I20260926 01:00:10.966739 12071 leader_election.cc:302] T 00000000000000000000000000000000 P f3224f7f89c54556bb862f8108f9bfd3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3224f7f89c54556bb862f8108f9bfd3; no voters:
I20260926 01:00:10.966929 12071 leader_election.cc:288] T 00000000000000000000000000000000 P f3224f7f89c54556bb862f8108f9bfd3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 01:00:10.966964 12075 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3224f7f89c54556bb862f8108f9bfd3 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 01:00:10.967070 12075 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3224f7f89c54556bb862f8108f9bfd3 [term 2 LEADER]: Becoming Leader. State: Replica: f3224f7f89c54556bb862f8108f9bfd3, State: Running, Role: LEADER
I20260926 01:00:10.967150 12075 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3224f7f89c54556bb862f8108f9bfd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3224f7f89c54556bb862f8108f9bfd3" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 45681 } }
I20260926 01:00:10.967206 12071 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3224f7f89c54556bb862f8108f9bfd3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:00:10.967418 12077 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3224f7f89c54556bb862f8108f9bfd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f3224f7f89c54556bb862f8108f9bfd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3224f7f89c54556bb862f8108f9bfd3" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 45681 } } }
I20260926 01:00:10.967507 12077 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3224f7f89c54556bb862f8108f9bfd3 [sys.catalog]: This master's current role is: LEADER
I20260926 01:00:10.967792 12076 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3224f7f89c54556bb862f8108f9bfd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3224f7f89c54556bb862f8108f9bfd3. Latest consensus state: current_term: 2 leader_uuid: "f3224f7f89c54556bb862f8108f9bfd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3224f7f89c54556bb862f8108f9bfd3" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 45681 } } }
I20260926 01:00:10.967895 12079 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:00:10.968389 12079 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:00:10.968549 12079 catalog_manager.cc:1259] Loaded cluster ID: 1d7837b5e3f04e0e91a4a4f58478383f
I20260926 01:00:10.968807 12079 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:00:10.970096 12079 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:00:10.967891 12076 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3224f7f89c54556bb862f8108f9bfd3 [sys.catalog]: This master's current role is: LEADER
I20260926 01:00:10.970263 12079 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f3224f7f89c54556bb862f8108f9bfd3: Loaded TSK: 0
I20260926 01:00:10.970496 12079 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 01:00:11.038009 12073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:00:11.038339 12073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:00:11.038365 12073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:00:11.039876 12073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:00:11.039925 12073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:00:11.039953 12073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.1
I20260926 01:00:11.041553 12073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:43391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/krb5kdc/oryx_127.3.75.1.keytab
--principal=oryx/127.3.75.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.1:33045
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.75.1
--webserver_port=42507
--tserver_master_addrs=127.3.75.62:45681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.1
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:00:11.041796 12073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:00:11.042049 12073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:00:11.047981 12097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:00:11.048032 12098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:00:11.048002 12100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:00:11.048272 12073 server_base.cc:1034] running on GCE node
I20260926 01:00:11.048552 12073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:00:11.048859 12073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:00:11.050019 12073 hybrid_clock.cc:648] HybridClock initialized: now 1790384411050007 us; error 27 us; skew 500 ppm
Sep 26 01:00:11 dist-test-slave-7v10 krb5kdc[11490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384411, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.75.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:00:11.052616 12073 init.cc:377] Logged in from keytab as oryx/127.3.75.1@KRBTEST.COM (short username oryx)
I20260926 01:00:11.053134 12073 webserver.cc:460] Webserver started at http://127.3.75.1:42507/ using document root <none> and password file <none>
I20260926 01:00:11.053359 12073 fs_manager.cc:359] Metadata directory not provided
I20260926 01:00:11.053447 12073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:00:11.054949 12073 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:00:11.055627 12107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:11.055819 12073 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:00:11.055926 12073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-0/wal
uuid: "c852054a20f3429b92b160cadcfd4bbb"
format_stamp: "Formatted at 2026-09-26 01:00:10 on dist-test-slave-7v10"
server_key: "338e42623a8d9c33d68dc877adbc229d"
server_key_iv: "588cebd306e14b095dcaf208907732f0"
server_key_version: "encryptionkey@0"
I20260926 01:00:11.056257 12073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:00:11.073702 12073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:00:11.075901 12073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:00:11.076051 12073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:00:11.076339 12073 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:00:11.076692 12073 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:00:11.076730 12073 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:11.076752 12073 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:00:11.076766 12073 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:11.084132 12073 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.1:33045
I20260926 01:00:11.084199 12220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.1:33045 every 8 connection(s)
I20260926 01:00:11.084578 12073 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
I20260926 01:00:11.089576 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 12073
I20260926 01:00:11.089903 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.2:40981
--local_ip_for_outbound_sockets=127.3.75.2
--tserver_master_addrs=127.3.75.62:45681
--webserver_port=38915
--webserver_interface=127.3.75.2
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab
--principal=kudu/127.3.75.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:43391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.75.2
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/krb5kdc/oryx_127.3.75.2.keytab with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 01:00:11 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.1@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:00:11 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.1@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:00:11.095007 12223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:11.085374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:45681 (local address 127.3.75.1:46911)
0926 01:00:11.085548 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:11.085607 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:11.086215 (+ 608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:11.086686 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:11.086690 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:11.087475 (+ 785us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:11.087806 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:11.087814 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:11.088917 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:11.088918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:11.090165 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:11.090168 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:11.091031 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:11.091032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:11.091171 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:11.091885 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:11.091895 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:11.094724 (+ 2829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.75.62:45681: Server oryx/127.3.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":81,"threads_started":1}
W20260926 01:00:11.095037 12224 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:00:11.085449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:46911 (local address 127.3.75.62:45681)
0926 01:00:11.085631 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:11.085639 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:11.085686 (+ 47us) server_negotiation.cc:407] Connection header received
0926 01:00:11.086426 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:11.086432 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:11.086473 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:11.086583 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:11.087939 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:11.088805 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:11.090259 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:11.090939 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:11.091048 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:11.094858 (+ 3810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.75.1:46911: BlockingRecv error: recv got EOF from 127.3.75.1:46911 (error 108)
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":74,"threads_started":1}
W20260926 01:00:11.095292 12221 heartbeater.cc:644] Failed to heartbeat to 127.3.75.62:45681 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.75.62:45681: Client connection negotiation failed: client connection to 127.3.75.62:45681: Server oryx/127.3.75.62@KRBTEST.COM not found in Kerberos database
Sep 26 01:00:11 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.1@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:00:11 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.1@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:00:11.101073 12226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:11.095513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:45681 (local address 127.3.75.1:50813)
0926 01:00:11.095671 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:11.095683 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:11.095752 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:11.095926 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:11.095926 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:11.096082 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:11.096132 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:11.096134 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:11.097111 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:11.097112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:11.097772 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:11.097774 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:11.098310 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:11.098311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:11.098354 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:11.098842 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:11.098846 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:11.100962 (+ 2116us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.75.62:45681: Server oryx/127.3.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":82,"threads_started":1}
W20260926 01:00:11.101089 12224 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:00:11.095678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:50813 (local address 127.3.75.62:45681)
0926 01:00:11.095735 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:11.095736 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:11.095745 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:00:11.095831 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:11.095832 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:11.095844 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:11.095913 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:11.096220 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:11.097013 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:11.097866 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:11.098214 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:11.098296 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:11.101046 (+ 2750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.75.1:50813: BlockingRecv error: recv got EOF from 127.3.75.1:50813 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 26 01:00:11 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.1@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:00:11 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.1@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:00:11.105691 12223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:11.101330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:45681 (local address 127.3.75.1:34079)
0926 01:00:11.101383 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:11.101397 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:11.101466 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:11.101619 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:11.101620 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:11.101766 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:11.101805 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:11.101807 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:11.102419 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:11.102420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:11.102983 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:11.102984 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:11.103351 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:11.103352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:11.103421 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:11.103858 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:11.103862 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:11.105622 (+ 1760us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.75.62:45681: Server oryx/127.3.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 01:00:11.105785 12224 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:00:11.101390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:34079 (local address 127.3.75.62:45681)
0926 01:00:11.101450 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:11.101452 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:11.101461 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:00:11.101526 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:11.101527 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:11.101537 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:11.101615 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:11.101888 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:11.102333 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:11.103081 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:11.103260 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:11.103349 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:11.105710 (+ 2361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.75.1:34079: BlockingRecv error: recv got EOF from 127.3.75.1:34079 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260926 01:00:11.105818 12221 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 01:00:11.174377 12225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:00:11.174556 12225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:00:11.174577 12225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:00:11.175697 12225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:00:11.175747 12225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:00:11.175773 12225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.2
I20260926 01:00:11.177081 12225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:43391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/krb5kdc/oryx_127.3.75.2.keytab
--principal=oryx/127.3.75.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.2:40981
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.75.2
--webserver_port=38915
--tserver_master_addrs=127.3.75.62:45681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.2
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:00:11.177268 12225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:00:11.177448 12225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:00:11.182436 12232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:00:11.182446 12225 server_base.cc:1034] running on GCE node
W20260926 01:00:11.182420 12234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:00:11.182420 12231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:00:11.182858 12225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:00:11.183172 12225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:00:11.184322 12225 hybrid_clock.cc:648] HybridClock initialized: now 1790384411184314 us; error 33 us; skew 500 ppm
Sep 26 01:00:11 dist-test-slave-7v10 krb5kdc[11490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384411, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.75.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:00:11.186717 12225 init.cc:377] Logged in from keytab as oryx/127.3.75.2@KRBTEST.COM (short username oryx)
I20260926 01:00:11.187170 12225 webserver.cc:460] Webserver started at http://127.3.75.2:38915/ using document root <none> and password file <none>
I20260926 01:00:11.187408 12225 fs_manager.cc:359] Metadata directory not provided
I20260926 01:00:11.187462 12225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:00:11.188795 12225 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:00:11.189369 12241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:11.189545 12225 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 01:00:11.189639 12225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-1/wal
uuid: "b72d4ac549cb474b9f939575bc51f61b"
format_stamp: "Formatted at 2026-09-26 01:00:10 on dist-test-slave-7v10"
server_key: "e3e2d0d75e2e12f45c24823fbd3ca3da"
server_key_iv: "9f5939b46fd0b6a3ca24dcec5f3c9bca"
server_key_version: "encryptionkey@0"
I20260926 01:00:11.189973 12225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:00:11.214519 12225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:00:11.216482 12225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:00:11.216658 12225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:00:11.216919 12225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:00:11.217235 12225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:00:11.217295 12225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:11.217329 12225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:00:11.217360 12225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:11.223258 12225 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.2:40981
I20260926 01:00:11.223309 12354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.2:40981 every 8 connection(s)
I20260926 01:00:11.223637 12225 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
I20260926 01:00:11.225725 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 12225
I20260926 01:00:11.226042 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.3:44383
--local_ip_for_outbound_sockets=127.3.75.3
--tserver_master_addrs=127.3.75.62:45681
--webserver_port=35345
--webserver_interface=127.3.75.3
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab
--principal=kudu/127.3.75.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:43391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.75.3
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/krb5kdc/oryx_127.3.75.3.keytab with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 01:00:11 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.2@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:00:11 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.2@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:00:11.231686 12357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:11.224399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:45681 (local address 127.3.75.2:53105)
0926 01:00:11.224564 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:11.224599 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:11.225055 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:11.225268 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:11.225271 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:11.226065 (+ 794us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:11.226358 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:11.226365 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:11.227104 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:11.227105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:11.228024 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:11.228026 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:11.228496 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:11.228497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:11.228592 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:11.229115 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:11.229123 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:11.231544 (+ 2421us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.75.62:45681: Server oryx/127.3.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":86,"threads_started":1}
W20260926 01:00:11.231775 12224 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:00:11.224404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:53105 (local address 127.3.75.62:45681)
0926 01:00:11.224476 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:11.224478 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:11.224692 (+ 214us) server_negotiation.cc:407] Connection header received
0926 01:00:11.225135 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:11.225136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:11.225148 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:11.225208 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:11.226451 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:11.227017 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:11.228114 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:11.228408 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:11.228481 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:11.231685 (+ 3204us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.75.2:53105: BlockingRecv error: recv got EOF from 127.3.75.2:53105 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260926 01:00:11.231974 12355 heartbeater.cc:644] Failed to heartbeat to 127.3.75.62:45681 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.75.62:45681: Client connection negotiation failed: client connection to 127.3.75.62:45681: Server oryx/127.3.75.62@KRBTEST.COM not found in Kerberos database
Sep 26 01:00:11 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.2@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:00:11 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.2@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:00:11.236896 12357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:11.232210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:45681 (local address 127.3.75.2:60279)
0926 01:00:11.232252 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:11.232262 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:11.232293 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:11.232440 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:11.232441 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:11.232545 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:11.232585 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:11.232586 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:11.233514 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:11.233515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:11.234005 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:11.234006 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:11.234502 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:11.234503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:11.234549 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:11.235020 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:11.235024 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:11.236852 (+ 1828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.75.62:45681: Server oryx/127.3.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260926 01:00:11.237020 12224 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:00:11.232277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:60279 (local address 127.3.75.62:45681)
0926 01:00:11.232322 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:11.232324 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:11.232333 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:00:11.232354 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:11.232355 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:11.232366 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:11.232439 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:11.232659 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:11.233430 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:11.234090 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:11.234422 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:11.234501 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:11.236958 (+ 2457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.75.2:60279: BlockingRecv error: recv got EOF from 127.3.75.2:60279 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 26 01:00:11 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.2@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:00:11 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.2@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:00:11.241832 12357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:11.237151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:45681 (local address 127.3.75.2:51947)
0926 01:00:11.237293 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:11.237303 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:11.237352 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:11.237484 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:11.237484 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:11.237584 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:11.237608 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:11.237609 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:11.238467 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:11.238468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:11.238995 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:11.238997 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:11.239468 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:11.239469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:11.239506 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:11.239937 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:11.239941 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:11.241767 (+ 1826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.75.62:45681: Server oryx/127.3.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":87,"threads_started":1}
W20260926 01:00:11.241925 12224 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:00:11.237212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:51947 (local address 127.3.75.62:45681)
0926 01:00:11.237239 (+ 27us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:11.237241 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:11.237389 (+ 148us) server_negotiation.cc:407] Connection header received
0926 01:00:11.237410 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:11.237411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:11.237419 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:11.237468 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:11.237680 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:11.238394 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:11.239068 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:11.239397 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:11.239460 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:11.241846 (+ 2386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.75.2:51947: BlockingRecv error: recv got EOF from 127.3.75.2:51947 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
W20260926 01:00:11.241981 12355 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 01:00:11.308733 12358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:00:11.308926 12358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:00:11.308949 12358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:00:11.310029 12358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:00:11.310063 12358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:00:11.310088 12358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.3
I20260926 01:00:11.311426 12358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:43391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/krb5kdc/oryx_127.3.75.3.keytab
--principal=oryx/127.3.75.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.3:44383
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.75.3
--webserver_port=35345
--tserver_master_addrs=127.3.75.62:45681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.3
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:00:11.311599 12358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:00:11.311766 12358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:00:11.316754 12367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:00:11.316824 12364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:00:11.316877 12365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:00:11.316983 12358 server_base.cc:1034] running on GCE node
I20260926 01:00:11.317198 12358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:00:11.317451 12358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:00:11.318627 12358 hybrid_clock.cc:648] HybridClock initialized: now 1790384411318608 us; error 29 us; skew 500 ppm
Sep 26 01:00:11 dist-test-slave-7v10 krb5kdc[11490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384411, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.75.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:00:11.321467 12358 init.cc:377] Logged in from keytab as oryx/127.3.75.3@KRBTEST.COM (short username oryx)
I20260926 01:00:11.321898 12358 webserver.cc:460] Webserver started at http://127.3.75.3:35345/ using document root <none> and password file <none>
I20260926 01:00:11.322160 12358 fs_manager.cc:359] Metadata directory not provided
I20260926 01:00:11.322258 12358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:00:11.323695 12358 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:00:11.324404 12374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:11.324613 12358 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:00:11.324729 12358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-2/wal
uuid: "492ff811e66140739750d238cb27accc"
format_stamp: "Formatted at 2026-09-26 01:00:10 on dist-test-slave-7v10"
server_key: "c4e22a504efd354df51d55a0d5f1568a"
server_key_iv: "5a84304576a7f778f048994626d39e32"
server_key_version: "encryptionkey@0"
I20260926 01:00:11.325065 12358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:00:11.335916 12358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:00:11.338183 12358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:00:11.338371 12358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:00:11.338593 12358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:00:11.338969 12358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:00:11.339028 12358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:11.339071 12358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:00:11.339123 12358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:11.345048 12358 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.3:44383
I20260926 01:00:11.345098 12487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.3:44383 every 8 connection(s)
I20260926 01:00:11.345513 12358 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
I20260926 01:00:11.350976 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 12358
Sep 26 01:00:11 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.3@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:00:11 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.3@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 01:00:11.353279 12224 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:00:11.346375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.3:56845 (local address 127.3.75.62:45681)
0926 01:00:11.346466 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:11.346468 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:11.346622 (+ 154us) server_negotiation.cc:407] Connection header received
0926 01:00:11.347122 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:11.347123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:11.347135 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:11.347209 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:11.348106 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:11.348620 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:11.349626 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:11.349827 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:11.349888 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:11.353220 (+ 3332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.75.3:56845: BlockingRecv error: recv got EOF from 127.3.75.3:56845 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 01:00:11.353363 12490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:11.346339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:45681 (local address 127.3.75.3:56845)
0926 01:00:11.346492 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:11.346531 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:11.347024 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:11.347265 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:11.347269 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:11.347814 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:11.348004 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:11.348008 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:11.348727 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:11.348728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:11.349518 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:11.349520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:11.349909 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:11.349910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:11.350010 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:11.350614 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:11.350622 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:11.353159 (+ 2537us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.75.62:45681: Server oryx/127.3.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":83,"threads_started":1}
W20260926 01:00:11.353616 12488 heartbeater.cc:644] Failed to heartbeat to 127.3.75.62:45681 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.75.62:45681: Client connection negotiation failed: client connection to 127.3.75.62:45681: Server oryx/127.3.75.62@KRBTEST.COM not found in Kerberos database
Sep 26 01:00:11 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.3@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:00:11 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.3@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:00:11.358811 12490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:11.353886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:45681 (local address 127.3.75.3:41613)
0926 01:00:11.353917 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:11.353927 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:11.353965 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:11.354084 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:11.354085 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:11.354222 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:11.354387 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:11.354390 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:11.355191 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:11.355192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:11.356060 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:11.356063 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:11.356478 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:11.356479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:11.356555 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:11.356935 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:11.356939 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:11.358724 (+ 1785us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.75.62:45681: Server oryx/127.3.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":15}
W20260926 01:00:11.358870 12224 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:00:11.353947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.3:41613 (local address 127.3.75.62:45681)
0926 01:00:11.353989 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:11.353991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:11.354001 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:00:11.354021 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:11.354022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:11.354029 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:11.354073 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:11.354488 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:11.355109 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:11.356169 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:11.356375 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:11.356476 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:11.358800 (+ 2324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.75.3:41613: BlockingRecv error: recv got EOF from 127.3.75.3:41613 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 01:00:11.359368 12492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:11.351643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50392 (local address 127.3.75.62:45681)
0926 01:00:11.351761 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:11.351763 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:11.351769 (+ 6us) server_negotiation.cc:407] Connection header received
0926 01:00:11.351790 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:11.351790 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:11.351799 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:11.351886 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:11.352480 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:11.352981 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:11.354036 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:11.354223 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:11.354276 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:11.355998 (+ 1722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:11.356002 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:11.356002 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:00:11.356012 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:11.357734 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:11.358218 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:11.358221 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:11.358222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:11.358360 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:11.358569 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:11.358571 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:11.358571 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:11.358806 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:11.359035 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:11.359130 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 01:00:11.359279 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":67,"threads_started":1}
Sep 26 01:00:11 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.3@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:00:11 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.3@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:00:11.363617 12490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:11.359443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:45681 (local address 127.3.75.3:53139)
0926 01:00:11.359483 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:11.359493 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:11.359542 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:11.359789 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:11.359790 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:11.359935 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:11.359993 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:11.359995 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:11.360698 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:11.360699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:11.361195 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:11.361197 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:11.361532 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:11.361533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:11.361591 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:11.361911 (+ 320us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:11.361915 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:11.363576 (+ 1661us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.75.62:45681: Server oryx/127.3.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":25}
W20260926 01:00:11.363740 12492 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:00:11.359655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.3:53139 (local address 127.3.75.62:45681)
0926 01:00:11.359694 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:11.359695 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:11.359701 (+ 6us) server_negotiation.cc:407] Connection header received
0926 01:00:11.359719 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:11.359719 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:11.359727 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:11.359778 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:11.360084 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:11.360628 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:11.361274 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:11.361446 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:11.361494 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:11.363663 (+ 2169us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.75.3:53139: BlockingRecv error: recv got EOF from 127.3.75.3:53139 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260926 01:00:11.363778 12488 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 01:00:12 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.1@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:00:12 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.1@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:00:12.110726 12493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:12.106236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:45681 (local address 127.3.75.1:33489)
0926 01:00:12.106392 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:12.106406 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:12.106468 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:12.106599 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:12.106600 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:12.106764 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:12.106814 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:12.106815 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:12.107526 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:12.107526 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:12.108041 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:12.108043 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:12.108374 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:12.108375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:12.108415 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:12.108966 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:12.108970 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:12.110657 (+ 1687us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.75.62:45681: Server oryx/127.3.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":67,"threads_started":1}
W20260926 01:00:12.110859 12494 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:00:12.106284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:33489 (local address 127.3.75.62:45681)
0926 01:00:12.106455 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:12.106456 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:12.106465 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:00:12.106525 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:12.106525 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:12.106534 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:12.106588 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:12.106927 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:12.107449 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:12.108116 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:12.108303 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:12.108353 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:12.110769 (+ 2416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.75.1:33489: BlockingRecv error: recv got EOF from 127.3.75.1:33489 (error 108)
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":94,"threads_started":1}
Sep 26 01:00:12 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.2@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:00:12 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.2@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:00:12.247597 12495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:12.242354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:45681 (local address 127.3.75.2:47671)
0926 01:00:12.242486 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:12.242497 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:12.242550 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:12.242674 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:12.242675 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:12.242850 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:12.242949 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:12.242951 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:12.243639 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:12.243641 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:12.244291 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:12.244294 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:12.244812 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:12.244813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:12.244902 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:12.245518 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:12.245523 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:12.247521 (+ 1998us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.75.62:45681: Server oryx/127.3.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":80,"threads_started":1}
W20260926 01:00:12.247658 12494 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:00:12.242425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:47671 (local address 127.3.75.62:45681)
0926 01:00:12.242473 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:12.242475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:12.242541 (+ 66us) server_negotiation.cc:407] Connection header received
0926 01:00:12.242592 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:12.242592 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:12.242602 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:12.242658 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:12.243009 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:12.243545 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:12.244468 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:12.244684 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:12.244797 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:12.247614 (+ 2817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.75.2:47671: BlockingRecv error: recv got EOF from 127.3.75.2:47671 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 26 01:00:12 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.3@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:00:12 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.3@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:00:12.369273 12496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:12.364336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:45681 (local address 127.3.75.3:60973)
0926 01:00:12.364509 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:12.364523 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:12.364601 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:12.364760 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:12.364761 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:12.364940 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:12.364988 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:12.364989 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:12.365678 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:12.365679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:12.366249 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:12.366251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:12.366647 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:12.366648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:12.366704 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:12.367274 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:12.367279 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:12.369205 (+ 1926us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.75.62:45681: Server oryx/127.3.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":88,"threads_started":1}
W20260926 01:00:12.369379 12494 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:00:12.364387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.3:60973 (local address 127.3.75.62:45681)
0926 01:00:12.364458 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:12.364460 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:12.364604 (+ 144us) server_negotiation.cc:407] Connection header received
0926 01:00:12.364661 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:12.364662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:12.364678 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:12.364754 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:12.365095 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:12.365588 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:12.366345 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:12.366561 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:12.366649 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:12.369308 (+ 2659us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.75.3:60973: BlockingRecv error: recv got EOF from 127.3.75.3:60973 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 26 01:00:13 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.1@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:00:13 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.1@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:00:13.115758 12497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:13.111245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:45681 (local address 127.3.75.1:44797)
0926 01:00:13.111368 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:13.111385 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:13.111432 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:13.111622 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:13.111623 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:13.111774 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:13.111824 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:13.111825 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:13.112429 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:13.112430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:13.112958 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:13.112960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:13.113276 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:13.113277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:13.113315 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:13.113760 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:13.113764 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:13.115683 (+ 1919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.75.62:45681: Server oryx/127.3.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":75,"threads_started":1}
W20260926 01:00:13.115825 12498 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:00:13.111356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:44797 (local address 127.3.75.62:45681)
0926 01:00:13.111494 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:13.111498 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:13.111510 (+ 12us) server_negotiation.cc:407] Connection header received
0926 01:00:13.111530 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:13.111531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:13.111541 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:13.111607 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:13.111885 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:13.112361 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:13.113029 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:13.113211 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:13.113260 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:13.115769 (+ 2509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.75.1:44797: BlockingRecv error: recv got EOF from 127.3.75.1:44797 (error 108)
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":94,"threads_started":1}
Sep 26 01:00:13 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.2@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:00:13 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.2@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:00:13.252882 12499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:13.248150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:45681 (local address 127.3.75.2:48799)
0926 01:00:13.248335 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:13.248345 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:13.248431 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:13.248611 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:13.248612 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:13.248849 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:13.248914 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:13.248917 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:13.249563 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:13.249563 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:13.250098 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:13.250099 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:13.250457 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:13.250458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:13.250498 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:13.250997 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:13.251007 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:13.252808 (+ 1801us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.75.62:45681: Server oryx/127.3.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":108,"threads_started":1}
W20260926 01:00:13.253036 12498 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:00:13.248272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:48799 (local address 127.3.75.62:45681)
0926 01:00:13.248326 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:13.248328 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:13.248414 (+ 86us) server_negotiation.cc:407] Connection header received
0926 01:00:13.248499 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:13.248500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:13.248513 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:13.248567 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:13.248991 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:13.249488 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:13.250177 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:13.250385 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:13.250439 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:13.252952 (+ 2513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.75.2:48799: BlockingRecv error: recv got EOF from 127.3.75.2:48799 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 26 01:00:13 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.3@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:00:13 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.3@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:00:13.374362 12500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:13.369737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:45681 (local address 127.3.75.3:44589)
0926 01:00:13.369875 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:13.369887 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:13.369970 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:13.370106 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:13.370106 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:13.370295 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:13.370343 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:13.370345 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:13.371151 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:13.371152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:13.371694 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:13.371696 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:13.372027 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:13.372028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:13.372066 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:13.372480 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:13.372485 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:13.374299 (+ 1814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.75.62:45681: Server oryx/127.3.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":54,"threads_started":1}
W20260926 01:00:13.374565 12498 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:00:13.369889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.3:44589 (local address 127.3.75.62:45681)
0926 01:00:13.369949 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:13.369951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:13.369961 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:00:13.370033 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:13.370034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:13.370048 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:13.370100 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:13.370435 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:13.371057 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:13.371771 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:13.371954 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:13.372005 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:13.374467 (+ 2462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.75.3:44589: BlockingRecv error: recv got EOF from 127.3.75.3:44589 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 26 01:00:14 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.1@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:00:14 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.1@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:00:14.120993 12501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:14.116276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:45681 (local address 127.3.75.1:59991)
0926 01:00:14.116409 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:14.116428 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:14.116473 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:14.116639 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:14.116640 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:14.116805 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:14.116855 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:14.116857 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:14.117482 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:14.117483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:14.117987 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:14.117989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:14.118322 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:14.118323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:14.118362 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:14.118931 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:14.118937 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:14.120937 (+ 2000us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.75.62:45681: Server oryx/127.3.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":82,"threads_started":1}
W20260926 01:00:14.121174 12502 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:00:14.116366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:59991 (local address 127.3.75.62:45681)
0926 01:00:14.116520 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:14.116521 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:14.116528 (+ 7us) server_negotiation.cc:407] Connection header received
0926 01:00:14.116549 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:14.116549 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:14.116559 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:14.116620 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:14.116924 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:14.117415 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:14.118061 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:14.118253 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:14.118304 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:14.121059 (+ 2755us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.75.1:59991: BlockingRecv error: recv got EOF from 127.3.75.1:59991 (error 108)
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":100,"threads_started":1}
Sep 26 01:00:14 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.2@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:00:14 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.2@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:00:14.257755 12503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:14.253398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:45681 (local address 127.3.75.2:52797)
0926 01:00:14.253526 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:14.253539 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:14.253611 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:14.253774 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:14.253775 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:14.253936 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:14.253984 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:14.253986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:14.254626 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:14.254626 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:14.255154 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:14.255156 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:14.255500 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:14.255501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:14.255538 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:14.255984 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:14.255988 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:14.257690 (+ 1702us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.75.62:45681: Server oryx/127.3.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":82,"threads_started":1}
W20260926 01:00:14.257827 12502 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:00:14.253476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:52797 (local address 127.3.75.62:45681)
0926 01:00:14.253521 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:14.253523 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:14.253599 (+ 76us) server_negotiation.cc:407] Connection header received
0926 01:00:14.253669 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:14.253669 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:14.253680 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:14.253722 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:14.254061 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:14.254557 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:14.255234 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:14.255431 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:14.255483 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:14.257781 (+ 2298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.75.2:52797: BlockingRecv error: recv got EOF from 127.3.75.2:52797 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 26 01:00:14 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.3@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:00:14 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.3@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:00:14.379791 12504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:14.374857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:45681 (local address 127.3.75.3:48193)
0926 01:00:14.375075 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:14.375086 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:14.375187 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:14.375322 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:14.375323 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:14.375502 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:14.375547 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:14.375549 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:14.376234 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:14.376235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:14.376787 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:14.376789 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:14.377185 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:14.377186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:14.377246 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:14.377744 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:14.377750 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:14.379693 (+ 1943us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.75.62:45681: Server oryx/127.3.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":128,"threads_started":1}
W20260926 01:00:14.379855 12502 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:00:14.374965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.3:48193 (local address 127.3.75.62:45681)
0926 01:00:14.375054 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:14.375056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:14.375167 (+ 111us) server_negotiation.cc:407] Connection header received
0926 01:00:14.375243 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:14.375244 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:14.375259 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:14.375307 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:14.375644 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:14.376136 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:14.376888 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:14.377091 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:14.377183 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:14.379786 (+ 2603us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.75.3:48193: BlockingRecv error: recv got EOF from 127.3.75.3:48193 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 26 01:00:15 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.1@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:00:15 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.1@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:00:15.126336 12505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:15.121439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:45681 (local address 127.3.75.1:56939)
0926 01:00:15.121576 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:15.121586 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:15.121636 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:15.121909 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:15.121910 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:15.122067 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:15.122117 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:15.122119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:15.122800 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:15.122801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:15.123378 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:15.123380 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:15.123720 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:15.123721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:15.123761 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:15.124194 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:15.124198 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:15.126219 (+ 2021us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.75.62:45681: Server oryx/127.3.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":76,"threads_started":1}
W20260926 01:00:15.126386 12506 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:00:15.121605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:56939 (local address 127.3.75.62:45681)
0926 01:00:15.121766 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:15.121768 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:15.121778 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:00:15.121809 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:15.121810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:15.121821 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:15.121893 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:15.122220 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:15.122728 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:15.123456 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:15.123652 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:15.123705 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:15.126309 (+ 2604us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.75.1:56939: BlockingRecv error: recv got EOF from 127.3.75.1:56939 (error 108)
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":105,"threads_started":1}
Sep 26 01:00:15 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.2@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:00:15 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.2@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:00:15.268319 12507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:15.258325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:45681 (local address 127.3.75.2:49887)
0926 01:00:15.258594 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:15.258608 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:15.258689 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:15.258963 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:15.258964 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:15.259191 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:15.259241 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:15.259242 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:15.259981 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:15.259982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:15.260524 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:15.260525 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:15.260895 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:15.260896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:15.260944 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:15.266054 (+ 5110us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:15.266059 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:15.268257 (+ 2198us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.75.62:45681: Server oryx/127.3.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":165,"threads_started":1}
W20260926 01:00:15.268477 12506 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:00:15.258428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:49887 (local address 127.3.75.62:45681)
0926 01:00:15.258497 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:15.258499 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:15.258666 (+ 167us) server_negotiation.cc:407] Connection header received
0926 01:00:15.258741 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:15.258742 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:15.258773 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:15.258842 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:15.259313 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:15.259897 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:15.260599 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:15.260803 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:15.260856 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:15.268380 (+ 7524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.75.2:49887: BlockingRecv error: recv got EOF from 127.3.75.2:49887 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 26 01:00:15 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.3@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:00:15 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.3@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:00:15.384989 12508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:15.380344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:45681 (local address 127.3.75.3:51843)
0926 01:00:15.380489 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:15.380506 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:15.380557 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:15.380704 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:15.380705 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:15.380883 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:15.380931 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:15.380933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:15.381560 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:15.381560 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:15.382091 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:15.382093 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:15.382420 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:15.382421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:15.382461 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:15.383017 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:15.383022 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:15.384888 (+ 1866us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.75.62:45681: Server oryx/127.3.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":91,"threads_started":1}
W20260926 01:00:15.385038 12506 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:00:15.380477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.3:51843 (local address 127.3.75.62:45681)
0926 01:00:15.380556 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:15.380558 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:15.380567 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:00:15.380597 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:15.380597 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:15.380609 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:15.380680 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:15.381002 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:15.381486 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:15.382172 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:15.382351 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:15.382403 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:15.384973 (+ 2570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.75.3:51843: BlockingRecv error: recv got EOF from 127.3.75.3:51843 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 26 01:00:16 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.1@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:00:16 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.1@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:00:16.131780 12509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:16.127001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:45681 (local address 127.3.75.1:36009)
0926 01:00:16.127176 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:16.127187 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:16.127247 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:16.127381 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:16.127382 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:16.127589 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:16.127651 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:16.127653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:16.128534 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:16.128535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:16.129032 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:16.129033 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:16.129414 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:16.129415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:16.129478 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:16.129903 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:16.129907 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:16.131725 (+ 1818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.75.62:45681: Server oryx/127.3.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":103,"threads_started":1}
W20260926 01:00:16.131924 12510 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:00:16.127065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:36009 (local address 127.3.75.62:45681)
0926 01:00:16.127192 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:16.127193 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:16.127233 (+ 40us) server_negotiation.cc:407] Connection header received
0926 01:00:16.127289 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:16.127290 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:16.127299 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:16.127360 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:16.127705 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:16.128459 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:16.129095 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:16.129315 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:16.129413 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:16.131836 (+ 2423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.75.1:36009: BlockingRecv error: recv got EOF from 127.3.75.1:36009 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":72,"threads_started":1}
Sep 26 01:00:16 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.2@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:00:16 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.2@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:00:16.273699 12511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:16.268850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:45681 (local address 127.3.75.2:33657)
0926 01:00:16.268981 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:16.268997 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:16.269068 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:16.269246 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:16.269248 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:16.269472 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:16.269520 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:16.269522 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:16.270377 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:16.270379 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:16.270931 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:16.270933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:16.271325 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:16.271326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:16.271383 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:16.271875 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:16.271880 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:16.273597 (+ 1717us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.75.62:45681: Server oryx/127.3.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":82,"threads_started":1}
W20260926 01:00:16.273737 12510 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:00:16.269016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:33657 (local address 127.3.75.62:45681)
0926 01:00:16.269090 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:16.269092 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:16.269113 (+ 21us) server_negotiation.cc:407] Connection header received
0926 01:00:16.269146 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:16.269147 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:16.269164 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:16.269225 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:16.269602 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:16.270286 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:16.271031 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:16.271237 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:16.271323 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:16.273678 (+ 2355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.75.2:33657: BlockingRecv error: recv got EOF from 127.3.75.2:33657 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 26 01:00:16 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.3@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:00:16 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.3@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:00:16.390198 12512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:16.385485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:45681 (local address 127.3.75.3:35183)
0926 01:00:16.385616 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:16.385627 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:16.385684 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:16.385824 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:16.385825 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:16.385975 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:16.386022 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:16.386024 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:16.386634 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:16.386635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:16.387210 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:16.387212 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:16.387529 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:16.387530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:16.387569 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:16.387972 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:16.387976 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:16.390127 (+ 2151us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.75.62:45681: Server oryx/127.3.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":83,"threads_started":1}
W20260926 01:00:16.390328 12510 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:00:16.385590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.3:35183 (local address 127.3.75.62:45681)
0926 01:00:16.385673 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:16.385675 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:16.385685 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:00:16.385741 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:16.385741 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:16.385753 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:16.385806 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:16.386089 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:16.386566 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:16.387284 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:16.387460 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:16.387513 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:16.390222 (+ 2709us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.75.3:35183: BlockingRecv error: recv got EOF from 127.3.75.3:35183 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 26 01:00:17 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.1@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:00:17 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.1@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:00:17.137001 12513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:17.132208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:45681 (local address 127.3.75.1:54155)
0926 01:00:17.132374 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:17.132387 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:17.132457 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:17.132667 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:17.132668 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:17.132836 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:17.132885 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:17.132887 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:17.133612 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:17.133613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:17.134130 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:17.134131 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:17.134587 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:17.134590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:17.134640 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:17.135117 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:17.135121 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:17.136899 (+ 1778us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.75.62:45681: Server oryx/127.3.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":63,"threads_started":1}
W20260926 01:00:17.137051 12514 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:00:17.132339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:54155 (local address 127.3.75.62:45681)
0926 01:00:17.132520 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:17.132522 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:17.132533 (+ 11us) server_negotiation.cc:407] Connection header received
0926 01:00:17.132560 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:17.132561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:17.132574 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:17.132642 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:17.132967 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:17.133542 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:17.134199 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:17.134499 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:17.134563 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:17.136979 (+ 2416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.75.1:54155: BlockingRecv error: recv got EOF from 127.3.75.1:54155 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":96,"threads_started":1}
Sep 26 01:00:17 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.2@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:00:17 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.2@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:00:17.279062 12515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:17.274312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:45681 (local address 127.3.75.2:48411)
0926 01:00:17.274474 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:17.274485 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:17.274555 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:17.274703 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:17.274704 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:17.274916 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:17.274964 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:17.274966 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:17.275619 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:17.275620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:17.276178 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:17.276180 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:17.276544 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:17.276545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:17.276605 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:17.277182 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:17.277186 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:17.279003 (+ 1817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.75.62:45681: Server oryx/127.3.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":84,"threads_started":1}
W20260926 01:00:17.279181 12514 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:00:17.274326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:48411 (local address 127.3.75.62:45681)
0926 01:00:17.274389 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:17.274391 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:17.274552 (+ 161us) server_negotiation.cc:407] Connection header received
0926 01:00:17.274611 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:17.274612 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:17.274624 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:17.274702 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:17.275057 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:17.275537 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:17.276273 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:17.276469 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:17.276547 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:17.279101 (+ 2554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.75.2:48411: BlockingRecv error: recv got EOF from 127.3.75.2:48411 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 26 01:00:17 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.3@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:00:17 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.3@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:00:17.395556 12516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:17.390781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:45681 (local address 127.3.75.3:41831)
0926 01:00:17.390963 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:17.390974 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:17.391053 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:17.391189 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:17.391190 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:17.391344 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:17.391394 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:17.391395 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:17.391996 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:17.391997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:17.392853 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:17.392856 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:17.393300 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:17.393301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:17.393343 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:17.393780 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:17.393784 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:17.395500 (+ 1716us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.75.62:45681: Server oryx/127.3.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":82,"threads_started":1}
W20260926 01:00:17.395737 12514 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:00:17.390923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.3:41831 (local address 127.3.75.62:45681)
0926 01:00:17.391001 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:17.391003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:17.391050 (+ 47us) server_negotiation.cc:407] Connection header received
0926 01:00:17.391097 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:17.391098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:17.391111 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:17.391165 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:17.391452 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:17.391932 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:17.392942 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:17.393222 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:17.393287 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:17.395647 (+ 2360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.75.3:41831: BlockingRecv error: recv got EOF from 127.3.75.3:41831 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 26 01:00:18 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.1@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:00:18 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.1@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:00:18.142426 12517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:18.137571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:45681 (local address 127.3.75.1:34335)
0926 01:00:18.137703 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:18.137755 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:18.137817 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:18.137988 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:18.137989 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:18.138196 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:18.138253 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:18.138256 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:18.139030 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:18.139031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:18.139640 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:18.139643 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:18.140068 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:18.140069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:18.140109 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:18.140592 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:18.140597 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:18.142369 (+ 1772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.75.62:45681: Server oryx/127.3.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":81,"threads_started":1}
W20260926 01:00:18.142572 12518 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:00:18.137645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:34335 (local address 127.3.75.62:45681)
0926 01:00:18.137796 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:18.137797 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:18.137806 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:00:18.137881 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:18.137882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:18.137895 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:18.137962 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:18.138340 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:18.138962 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:18.139727 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:18.139979 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:18.140066 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:18.142466 (+ 2400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.75.1:34335: BlockingRecv error: recv got EOF from 127.3.75.1:34335 (error 108)
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":93,"threads_started":1}
Sep 26 01:00:18 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.2@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:00:18 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.2@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:00:18.284144 12519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:18.279537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:45681 (local address 127.3.75.2:36257)
0926 01:00:18.279681 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:18.279691 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:18.279751 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:18.279868 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:18.279869 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:18.280061 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:18.280112 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:18.280113 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:18.280760 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:18.280762 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:18.281303 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:18.281305 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:18.281634 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:18.281635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:18.281674 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:18.282197 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:18.282202 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:18.284078 (+ 1876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.75.62:45681: Server oryx/127.3.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":80,"threads_started":1}
W20260926 01:00:18.284245 12518 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:00:18.279591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:36257 (local address 127.3.75.62:45681)
0926 01:00:18.279663 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:18.279665 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:18.279745 (+ 80us) server_negotiation.cc:407] Connection header received
0926 01:00:18.279802 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:18.279803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:18.279813 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:18.279863 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:18.280173 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:18.280663 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:18.281380 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:18.281566 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:18.281621 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:18.284177 (+ 2556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.75.2:36257: BlockingRecv error: recv got EOF from 127.3.75.2:36257 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 26 01:00:18 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.3@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:00:18 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.3@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:00:18.401212 12520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:18.396018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:45681 (local address 127.3.75.3:53323)
0926 01:00:18.396176 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:18.396186 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:18.396236 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:18.396383 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:18.396384 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:18.396537 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:18.396585 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:18.396587 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:18.397405 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:18.397406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:18.398095 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:18.398099 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:18.398501 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:18.398502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:18.398542 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:18.399049 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:18.399054 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:18.401146 (+ 2092us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.75.62:45681: Server oryx/127.3.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":76,"threads_started":1}
W20260926 01:00:18.401279 12518 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:00:18.396129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.3:53323 (local address 127.3.75.62:45681)
0926 01:00:18.396222 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:18.396224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:18.396245 (+ 21us) server_negotiation.cc:407] Connection header received
0926 01:00:18.396275 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:18.396276 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:18.396299 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:18.396368 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:18.396666 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:18.397317 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:18.398192 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:18.398400 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:18.398493 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:18.401232 (+ 2739us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.75.3:53323: BlockingRecv error: recv got EOF from 127.3.75.3:53323 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 26 01:00:19 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.1@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:00:19 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.1@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:00:19.147317 12521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:19.142978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:45681 (local address 127.3.75.1:48401)
0926 01:00:19.143112 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:19.143125 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:19.143195 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:19.143328 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:19.143328 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:19.143484 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:19.143535 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:19.143537 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:19.144180 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:19.144181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:19.144705 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:19.144707 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:19.145018 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:19.145019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:19.145057 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:19.145487 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:19.145492 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:19.147258 (+ 1766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.75.62:45681: Server oryx/127.3.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":69,"threads_started":1}
W20260926 01:00:19.147429 12522 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:00:19.143070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:48401 (local address 127.3.75.62:45681)
0926 01:00:19.143207 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:19.143209 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:19.143217 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:00:19.143239 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:19.143240 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:19.143249 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:19.143308 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:19.143607 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:19.144116 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:19.144774 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:19.144956 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:19.145004 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:19.147352 (+ 2348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.75.1:48401: BlockingRecv error: recv got EOF from 127.3.75.1:48401 (error 108)
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":76,"threads_started":1}
Sep 26 01:00:19 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.2@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:00:19 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.2@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:00:19.289211 12523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:19.284781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:45681 (local address 127.3.75.2:49785)
0926 01:00:19.284958 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:19.284969 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:19.285033 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:19.285162 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:19.285163 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:19.285322 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:19.285367 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:19.285369 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:19.286128 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:19.286129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:19.286630 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:19.286631 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:19.286940 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:19.286941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:19.286980 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:19.287469 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:19.287480 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:19.289155 (+ 1675us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.75.62:45681: Server oryx/127.3.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":118,"threads_started":1}
W20260926 01:00:19.289340 12522 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:00:19.284887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:49785 (local address 127.3.75.62:45681)
0926 01:00:19.284955 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:19.284957 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:19.285036 (+ 79us) server_negotiation.cc:407] Connection header received
0926 01:00:19.285076 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:19.285077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:19.285087 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:19.285147 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:19.285427 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:19.286061 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:19.286698 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:19.286890 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:19.286937 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:19.289260 (+ 2323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.75.2:49785: BlockingRecv error: recv got EOF from 127.3.75.2:49785 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 26 01:00:19 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.3@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:00:19 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.3@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:00:19.406263 12524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:19.401683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:45681 (local address 127.3.75.3:44611)
0926 01:00:19.401848 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:19.401859 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:19.401948 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:19.402097 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:19.402098 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:19.402286 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:19.402332 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:19.402333 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:19.403080 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:19.403081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:19.403609 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:19.403610 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:19.403979 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:19.403980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:19.404041 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:19.404492 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:19.404497 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:19.406166 (+ 1669us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.75.62:45681: Server oryx/127.3.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":87,"threads_started":1}
W20260926 01:00:19.406301 12522 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:00:19.401771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.3:44611 (local address 127.3.75.62:45681)
0926 01:00:19.401846 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:19.401848 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:19.401934 (+ 86us) server_negotiation.cc:407] Connection header received
0926 01:00:19.402006 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:19.402007 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:19.402021 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:19.402093 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:19.402428 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:19.402993 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:19.403705 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:19.403899 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:19.403969 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:19.406239 (+ 2270us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.75.3:44611: BlockingRecv error: recv got EOF from 127.3.75.3:44611 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 26 01:00:20 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.1@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:00:20 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.1@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:00:20.152300 12525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:20.147795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:45681 (local address 127.3.75.1:49435)
0926 01:00:20.147949 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:20.147959 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:20.148016 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:20.148164 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:20.148164 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:20.148325 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:20.148373 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:20.148375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:20.149010 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:20.149011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:20.149527 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:20.149528 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:20.149849 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:20.149850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:20.149887 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:20.150384 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:20.150389 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:20.152242 (+ 1853us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.75.62:45681: Server oryx/127.3.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":88,"threads_started":1}
W20260926 01:00:20.152412 12526 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:00:20.147885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:49435 (local address 127.3.75.62:45681)
0926 01:00:20.147989 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:20.147991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:20.148002 (+ 11us) server_negotiation.cc:407] Connection header received
0926 01:00:20.148075 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:20.148075 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:20.148085 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:20.148143 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:20.148440 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:20.148946 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:20.149596 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:20.149785 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:20.149838 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:20.152328 (+ 2490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.75.1:49435: BlockingRecv error: recv got EOF from 127.3.75.1:49435 (error 108)
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":48,"threads_started":1}
Sep 26 01:00:20 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.2@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:00:20 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.2@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:00:20.299829 12527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:20.289721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:45681 (local address 127.3.75.2:49721)
0926 01:00:20.289870 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:20.289884 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:20.289945 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:20.290085 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:20.290086 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:20.290245 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:20.290292 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:20.290294 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:20.290956 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:20.290956 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:20.291478 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:20.291479 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:20.291793 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:20.291794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:20.291838 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:20.297623 (+ 5785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:20.297628 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:20.299726 (+ 2098us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.75.62:45681: Server oryx/127.3.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":91,"threads_started":1}
W20260926 01:00:20.299894 12526 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:00:20.289819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:49721 (local address 127.3.75.62:45681)
0926 01:00:20.289891 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:20.289892 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:20.289935 (+ 43us) server_negotiation.cc:407] Connection header received
0926 01:00:20.290010 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:20.290011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:20.290020 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:20.290065 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:20.290356 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:20.290838 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:20.291549 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:20.291731 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:20.291780 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:20.299828 (+ 8048us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.75.2:49721: BlockingRecv error: recv got EOF from 127.3.75.2:49721 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 26 01:00:20 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.3@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:00:20 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.3@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:00:20.410957 12528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:20.406771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:45681 (local address 127.3.75.3:51611)
0926 01:00:20.406970 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:20.406981 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:20.407040 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:20.407155 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:20.407155 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:20.407307 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:20.407355 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:20.407356 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:20.407964 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:20.407965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:20.408489 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:20.408491 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:20.408784 (+ 293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:20.408784 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:20.408827 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:20.409226 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:20.409230 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:20.410842 (+ 1612us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.75.62:45681: Server oryx/127.3.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":127,"threads_started":1}
W20260926 01:00:20.411016 12526 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:00:20.406831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.3:51611 (local address 127.3.75.62:45681)
0926 01:00:20.406916 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:20.406918 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:20.407038 (+ 120us) server_negotiation.cc:407] Connection header received
0926 01:00:20.407085 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:20.407086 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:20.407099 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:20.407150 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:20.407409 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:20.407898 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:20.408551 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:20.408723 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:20.408770 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:20.410951 (+ 2181us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.75.3:51611: BlockingRecv error: recv got EOF from 127.3.75.3:51611 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 26 01:00:21 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.1@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:00:21 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.1@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:00:21.157639 12529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:21.152916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:45681 (local address 127.3.75.1:55889)
0926 01:00:21.153065 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:21.153075 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:21.153132 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:21.153260 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:21.153261 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:21.153417 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:21.153489 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:21.153491 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:21.154157 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:21.154158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:21.154693 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:21.154694 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:21.155162 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:21.155163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:21.155244 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:21.155788 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:21.155793 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:21.157568 (+ 1775us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.75.62:45681: Server oryx/127.3.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":84,"threads_started":1}
W20260926 01:00:21.157734 12530 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:00:21.152995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:55889 (local address 127.3.75.62:45681)
0926 01:00:21.153118 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:21.153120 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:21.153136 (+ 16us) server_negotiation.cc:407] Connection header received
0926 01:00:21.153176 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:21.153176 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:21.153185 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:21.153245 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:21.153558 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:21.154067 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:21.154761 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:21.155070 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:21.155136 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:21.157661 (+ 2525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.75.1:55889: BlockingRecv error: recv got EOF from 127.3.75.1:55889 (error 108)
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":81,"threads_started":1}
Sep 26 01:00:21 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.2@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:00:21 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.2@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:00:21.304669 12531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:21.300365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:45681 (local address 127.3.75.2:56445)
0926 01:00:21.300522 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:21.300533 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:21.300590 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:21.300730 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:21.300730 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:21.300886 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:21.300933 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:21.300934 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:21.301532 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:21.301532 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:21.302043 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:21.302044 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:21.302350 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:21.302351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:21.302391 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:21.302808 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:21.302812 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:21.304572 (+ 1760us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.75.62:45681: Server oryx/127.3.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":87,"threads_started":1}
W20260926 01:00:21.304720 12530 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:00:21.300423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:56445 (local address 127.3.75.62:45681)
0926 01:00:21.300476 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:21.300478 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:21.300578 (+ 100us) server_negotiation.cc:407] Connection header received
0926 01:00:21.300639 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:21.300639 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:21.300650 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:21.300711 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:21.300999 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:21.301470 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:21.302110 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:21.302288 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:21.302337 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:21.304659 (+ 2322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.75.2:56445: BlockingRecv error: recv got EOF from 127.3.75.2:56445 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 26 01:00:21 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.3@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:00:21 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.3@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:00:21.416502 12532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:21.411608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:45681 (local address 127.3.75.3:45457)
0926 01:00:21.411778 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:21.411791 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:21.411863 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:21.412058 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:21.412066 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:21.412251 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:21.412301 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:21.412303 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:21.412993 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:21.412995 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:21.413615 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:21.413617 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:21.414031 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:21.414033 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:21.414112 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:21.414615 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:21.414627 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:21.416443 (+ 1816us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.75.62:45681: Server oryx/127.3.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":89,"threads_started":1}
W20260926 01:00:21.416611 12530 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:00:21.411742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.3:45457 (local address 127.3.75.62:45681)
0926 01:00:21.411815 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:21.411817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:21.411843 (+ 26us) server_negotiation.cc:407] Connection header received
0926 01:00:21.411942 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:21.411943 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:21.411958 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:21.412043 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:21.412389 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:21.412887 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:21.413712 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:21.413957 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:21.414033 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:21.416531 (+ 2498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.75.3:45457: BlockingRecv error: recv got EOF from 127.3.75.3:45457 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 26 01:00:22 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.1@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:00:22 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.1@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:00:22.162698 12533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:22.158185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:45681 (local address 127.3.75.1:60639)
0926 01:00:22.158332 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:22.158342 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:22.158418 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:22.158585 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:22.158586 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:22.158742 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:22.158791 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:22.158793 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:22.159533 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:22.159534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:22.160058 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:22.160060 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:22.160369 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:22.160369 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:22.160406 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:22.160903 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:22.160908 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:22.162641 (+ 1733us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.75.62:45681: Server oryx/127.3.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":82,"threads_started":1}
W20260926 01:00:22.162807 12534 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:00:22.158274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:60639 (local address 127.3.75.62:45681)
0926 01:00:22.158399 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:22.158401 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:22.158410 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:00:22.158490 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:22.158490 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:22.158500 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:22.158558 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:22.158854 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:22.159461 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:22.160127 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:22.160308 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:22.160356 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:22.162728 (+ 2372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.75.1:60639: BlockingRecv error: recv got EOF from 127.3.75.1:60639 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":52,"threads_started":1}
Sep 26 01:00:22 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.2@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:00:22 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.2@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:00:22.309679 12535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:22.305193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:45681 (local address 127.3.75.2:34211)
0926 01:00:22.305327 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:22.305345 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:22.305401 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:22.305530 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:22.305531 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:22.305700 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:22.305747 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:22.305748 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:22.306545 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:22.306546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:22.307106 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:22.307108 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:22.307394 (+ 286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:22.307394 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:22.307432 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:22.307946 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:22.307951 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:22.309622 (+ 1671us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.75.62:45681: Server oryx/127.3.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":86,"threads_started":1}
W20260926 01:00:22.309800 12534 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:00:22.305235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:34211 (local address 127.3.75.62:45681)
0926 01:00:22.305303 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:22.305305 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:22.305401 (+ 96us) server_negotiation.cc:407] Connection header received
0926 01:00:22.305448 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:22.305449 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:22.305459 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:22.305515 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:22.305814 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:22.306478 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:22.307168 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:22.307337 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:22.307384 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:22.309705 (+ 2321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.75.2:34211: BlockingRecv error: recv got EOF from 127.3.75.2:34211 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 26 01:00:22 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.3@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:00:22 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.3@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:00:22.421808 12536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:22.417059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:45681 (local address 127.3.75.3:37221)
0926 01:00:22.417234 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:22.417252 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:22.417330 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:22.417502 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:22.417503 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:22.417692 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:22.417737 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:22.417738 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:22.418391 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:22.418392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:22.418931 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:22.418933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:22.419302 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:22.419303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:22.419367 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:22.419846 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:22.419851 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:22.421712 (+ 1861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.75.62:45681: Server oryx/127.3.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":87,"threads_started":1}
W20260926 01:00:22.421859 12534 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:00:22.417134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.3:37221 (local address 127.3.75.62:45681)
0926 01:00:22.417216 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:22.417219 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:22.417312 (+ 93us) server_negotiation.cc:407] Connection header received
0926 01:00:22.417419 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:22.417420 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:22.417432 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:22.417494 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:22.417819 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:22.418304 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:22.419015 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:22.419215 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:22.419300 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:22.421793 (+ 2493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.75.3:37221: BlockingRecv error: recv got EOF from 127.3.75.3:37221 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 26 01:00:23 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.1@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:00:23 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.1@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:00:23.167621 12537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:23.163129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:45681 (local address 127.3.75.1:58793)
0926 01:00:23.163252 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:23.163264 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:23.163318 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:23.163445 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:23.163445 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:23.163610 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:23.163661 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:23.163663 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:23.164290 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:23.164291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:23.164801 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:23.164802 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:23.165124 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:23.165125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:23.165165 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:23.165608 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:23.165612 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:23.167531 (+ 1919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.75.62:45681: Server oryx/127.3.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":71,"threads_started":1}
W20260926 01:00:23.167752 12538 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:00:23.163209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:58793 (local address 127.3.75.62:45681)
0926 01:00:23.163328 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:23.163329 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:23.163337 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:00:23.163360 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:23.163360 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:23.163370 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:23.163432 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:23.163726 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:23.164225 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:23.164878 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:23.165065 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:23.165114 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:23.167635 (+ 2521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.75.1:58793: BlockingRecv error: recv got EOF from 127.3.75.1:58793 (error 108)
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":67,"threads_started":1}
Sep 26 01:00:23 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.2@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:00:23 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.2@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:00:23.314420 12539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:23.310157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:45681 (local address 127.3.75.2:37461)
0926 01:00:23.310302 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:23.310313 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:23.310368 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:23.310482 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:23.310483 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:23.310657 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:23.310705 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:23.310707 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:23.311368 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:23.311369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:23.311892 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:23.311893 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:23.312207 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:23.312208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:23.312245 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:23.312662 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:23.312666 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:23.314349 (+ 1683us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.75.62:45681: Server oryx/127.3.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":85,"threads_started":1}
W20260926 01:00:23.314560 12538 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:00:23.310241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:37461 (local address 127.3.75.62:45681)
0926 01:00:23.310286 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:23.310288 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:23.310355 (+ 67us) server_negotiation.cc:407] Connection header received
0926 01:00:23.310423 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:23.310424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:23.310435 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:23.310470 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:23.310762 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:23.311307 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:23.311957 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:23.312148 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:23.312198 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:23.314480 (+ 2282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.75.2:37461: BlockingRecv error: recv got EOF from 127.3.75.2:37461 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 26 01:00:23 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.3@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:00:23 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.3@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:00:23.426945 12540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:23.422287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:45681 (local address 127.3.75.3:39309)
0926 01:00:23.422440 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:23.422458 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:23.422518 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:23.422639 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:23.422640 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:23.422795 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:23.422843 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:23.422845 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:23.423517 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:23.423518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:23.424060 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:23.424062 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:23.424354 (+ 292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:23.424355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:23.424393 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:23.424885 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:23.424890 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:23.426860 (+ 1970us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.75.62:45681: Server oryx/127.3.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":53,"threads_started":1}
W20260926 01:00:23.427068 12538 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:00:23.422396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.3:39309 (local address 127.3.75.62:45681)
0926 01:00:23.422469 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:23.422471 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:23.422515 (+ 44us) server_negotiation.cc:407] Connection header received
0926 01:00:23.422569 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:23.422570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:23.422582 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:23.422627 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:23.422963 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:23.423454 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:23.424127 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:23.424296 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:23.424343 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:23.426990 (+ 2647us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.75.3:39309: BlockingRecv error: recv got EOF from 127.3.75.3:39309 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 26 01:00:24 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.1@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:00:24 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.1@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:00:24.172437 12541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:24.168069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:45681 (local address 127.3.75.1:51199)
0926 01:00:24.168217 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:24.168228 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:24.168316 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:24.168443 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:24.168444 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:24.168613 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:24.168662 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:24.168663 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:24.169323 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:24.169324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:24.169851 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:24.169853 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:24.170166 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:24.170167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:24.170207 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:24.170688 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:24.170693 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:24.172374 (+ 1681us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.75.62:45681: Server oryx/127.3.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":67,"threads_started":1}
W20260926 01:00:24.172555 12542 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:00:24.168151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:51199 (local address 127.3.75.62:45681)
0926 01:00:24.168267 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:24.168269 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:24.168293 (+ 24us) server_negotiation.cc:407] Connection header received
0926 01:00:24.168358 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:24.168359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:24.168369 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:24.168425 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:24.168720 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:24.169258 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:24.169914 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:24.170107 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:24.170154 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:24.172457 (+ 2303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.75.1:51199: BlockingRecv error: recv got EOF from 127.3.75.1:51199 (error 108)
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":51,"threads_started":1}
Sep 26 01:00:24 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.2@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:00:24 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.2@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:00:24.319314 12543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:24.314932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:45681 (local address 127.3.75.2:33187)
0926 01:00:24.315096 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:24.315109 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:24.315177 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:24.315328 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:24.315329 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:24.315483 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:24.315530 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:24.315531 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:24.316140 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:24.316141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:24.316671 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:24.316673 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:24.317041 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:24.317042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:24.317105 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:24.317505 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:24.317509 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:24.319209 (+ 1700us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.75.62:45681: Server oryx/127.3.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":56,"threads_started":1}
W20260926 01:00:24.319355 12542 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:00:24.315153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:33187 (local address 127.3.75.62:45681)
0926 01:00:24.315215 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:24.315221 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:24.315229 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:00:24.315254 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:24.315255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:24.315264 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:24.315311 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:24.315589 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:24.316077 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:24.316758 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:24.316951 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:24.317034 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:24.319292 (+ 2258us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.75.2:33187: BlockingRecv error: recv got EOF from 127.3.75.2:33187 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 26 01:00:24 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.3@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:00:24 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.3@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:00:24.431891 12544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:24.427462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:45681 (local address 127.3.75.3:44575)
0926 01:00:24.427661 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:24.427671 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:24.427799 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:24.427945 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:24.427946 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:24.428106 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:24.428153 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:24.428154 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:24.428877 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:24.428878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:24.429426 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:24.429427 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:24.429729 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:24.429730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:24.429768 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:24.430181 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:24.430185 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:24.431824 (+ 1639us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.75.62:45681: Server oryx/127.3.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":116,"threads_started":1}
W20260926 01:00:24.431967 12542 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:00:24.427529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.3:44575 (local address 127.3.75.62:45681)
0926 01:00:24.427605 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:24.427607 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:24.427793 (+ 186us) server_negotiation.cc:407] Connection header received
0926 01:00:24.427872 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:24.427872 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:24.427886 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:24.427931 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:24.428226 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:24.428811 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:24.429491 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:24.429674 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:24.429720 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:24.431908 (+ 2188us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.75.3:44575: BlockingRecv error: recv got EOF from 127.3.75.3:44575 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 26 01:00:25 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.1@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:00:25 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.1@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:00:25.177445 12545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:25.172899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:45681 (local address 127.3.75.1:43953)
0926 01:00:25.173038 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:25.173050 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:25.173099 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:25.173325 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:25.173325 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:25.173486 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:25.173545 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:25.173546 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:25.174230 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:25.174231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:25.174786 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:25.174787 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:25.175164 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:25.175164 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:25.175211 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:25.175686 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:25.175690 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:25.177371 (+ 1681us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.75.62:45681: Server oryx/127.3.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":72,"threads_started":1}
W20260926 01:00:25.177532 12546 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:00:25.173043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:43953 (local address 127.3.75.62:45681)
0926 01:00:25.173193 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:25.173195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:25.173204 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:00:25.173229 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:25.173229 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:25.173240 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:25.173313 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:25.173640 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:25.174158 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:25.174857 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:25.175075 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:25.175152 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:25.177462 (+ 2310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.75.1:43953: BlockingRecv error: recv got EOF from 127.3.75.1:43953 (error 108)
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":93,"threads_started":1}
Sep 26 01:00:25 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.2@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:00:25 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.2@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:00:25.330113 12547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:25.319888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:45681 (local address 127.3.75.2:47371)
0926 01:00:25.320026 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:25.320039 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:25.320109 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:25.320385 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:25.320387 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:25.320679 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:25.320740 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:25.320743 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:25.321745 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:25.321746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:25.322708 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:25.322710 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:25.323200 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:25.323201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:25.323278 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:25.328361 (+ 5083us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:25.328366 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:25.330042 (+ 1676us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.75.62:45681: Server oryx/127.3.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":74,"threads_started":1}
W20260926 01:00:25.330246 12546 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:00:25.319959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:47371 (local address 127.3.75.62:45681)
0926 01:00:25.320008 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:25.320011 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:25.320097 (+ 86us) server_negotiation.cc:407] Connection header received
0926 01:00:25.320183 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:25.320185 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:25.320199 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:25.320286 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:25.320811 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:25.321661 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:25.322788 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:25.323118 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:25.323199 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:25.330162 (+ 6963us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.75.2:47371: BlockingRecv error: recv got EOF from 127.3.75.2:47371 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 26 01:00:25 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.3@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:00:25 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.3@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:00:25.436772 12548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:25.432291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:45681 (local address 127.3.75.3:50885)
0926 01:00:25.432456 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:25.432467 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:25.432550 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:25.432699 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:25.432700 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:25.432875 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:25.432921 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:25.432922 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:25.433571 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:25.433572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:25.434125 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:25.434127 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:25.434487 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:25.434487 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:25.434548 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:25.435008 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:25.435013 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:25.436662 (+ 1649us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.75.62:45681: Server oryx/127.3.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":86,"threads_started":1}
W20260926 01:00:25.436820 12546 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:00:25.432366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.3:50885 (local address 127.3.75.62:45681)
0926 01:00:25.432439 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:25.432441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:25.432533 (+ 92us) server_negotiation.cc:407] Connection header received
0926 01:00:25.432622 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:25.432623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:25.432638 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:25.432684 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:25.432998 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:25.433483 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:25.434209 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:25.434405 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:25.434485 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:25.436751 (+ 2266us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.75.3:50885: BlockingRecv error: recv got EOF from 127.3.75.3:50885 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 26 01:00:26 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.1@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:00:26 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.1@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:00:26.182632 12549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:26.178214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:45681 (local address 127.3.75.1:33103)
0926 01:00:26.178381 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:26.178394 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:26.178457 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:26.178675 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:26.178676 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:26.178911 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:26.178960 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:26.178962 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:26.179628 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:26.179628 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:26.180140 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:26.180141 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:26.180458 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:26.180459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:26.180497 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:26.180933 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:26.180937 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:26.182575 (+ 1638us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.75.62:45681: Server oryx/127.3.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":93,"threads_started":1}
W20260926 01:00:26.182758 12550 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:00:26.178299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:33103 (local address 127.3.75.62:45681)
0926 01:00:26.178394 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:26.178396 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:26.178444 (+ 48us) server_negotiation.cc:407] Connection header received
0926 01:00:26.178569 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:26.178570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:26.178585 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:26.178659 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:26.179023 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:26.179561 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:26.180212 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:26.180399 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:26.180449 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:26.182671 (+ 2222us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.75.1:33103: BlockingRecv error: recv got EOF from 127.3.75.1:33103 (error 108)
Metrics: {"server-negotiator.queue_time_us":73,"thread_start_us":47,"threads_started":1}
Sep 26 01:00:26 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.2@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:00:26 dist-test-slave-7v10 krb5kdc[11490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.75.2@KRBTEST.COM for oryx/127.3.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:00:26.334945 12551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:26.330593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:45681 (local address 127.3.75.2:37323)
0926 01:00:26.330703 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:26.330717 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:26.330770 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:26.330912 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:26.330912 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:26.331080 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:26.331129 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:26.331130 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:26.331721 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:26.331722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:26.332252 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:26.332254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:26.332604 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:26.332605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:26.332642 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:26.333051 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:26.333055 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:26.334826 (+ 1771us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.75.62:45681: Server oryx/127.3.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":64,"threads_started":1}
W20260926 01:00:26.335042 12550 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:00:26.330643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:37323 (local address 127.3.75.62:45681)
0926 01:00:26.330705 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:26.330707 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:26.330768 (+ 61us) server_negotiation.cc:407] Connection header received
0926 01:00:26.330813 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:26.330813 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:26.330823 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:26.330887 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:26.331185 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:26.331653 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:26.332347 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:26.332539 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:26.332597 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:26.334961 (+ 2364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.75.2:37323: BlockingRecv error: recv got EOF from 127.3.75.2:37323 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 01:00:26.361471 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 12073
I20260926 01:00:26.365882 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 12225
I20260926 01:00:26.370281 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 12358
I20260926 01:00:26.374298 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 12003
2026-09-26T01:00:26Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17438 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-26T01:00:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:00:26Z Disabled control of system clock
I20260926 01:00:26.401973 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:33735
--webserver_interface=127.3.75.62
--webserver_port=0
--builtin_ntp_servers=127.3.75.20:34359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.75.62:33735
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 01:00:26.481861 12555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:00:26.482071 12555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:00:26.482091 12555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:00:26.483309 12555 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:00:26.483350 12555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:00:26.483363 12555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:00:26.483376 12555 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:00:26.483429 12555 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:00:26.484724 12555 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:34359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.75.62:33735
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:33735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:00:26.484963 12555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:00:26.485248 12555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 01:00:26.490528 12555 server_base.cc:1034] running on GCE node
W20260926 01:00:26.490559 12561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:00:26.490579 12560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:00:26.490687 12563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:00:26.490967 12555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:00:26.491312 12555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:00:26.492486 12555 hybrid_clock.cc:648] HybridClock initialized: now 1790384426492465 us; error 33 us; skew 500 ppm
I20260926 01:00:26.493705 12555 webserver.cc:460] Webserver started at http://127.3.75.62:38223/ using document root <none> and password file <none>
I20260926 01:00:26.493943 12555 fs_manager.cc:359] Metadata directory not provided
I20260926 01:00:26.493986 12555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:00:26.494091 12555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:00:26.495155 12555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/master-0/data/instance:
uuid: "5c88c5f91e97487c812ac51bc9cc7c10"
format_stamp: "Formatted at 2026-09-26 01:00:26 on dist-test-slave-7v10"
server_key: "40e4fa4eb417b4b39d003521b8d0ce18"
server_key_iv: "ef988b518736b49f8bf18c3e81387918"
server_key_version: "encryptionkey@0"
I20260926 01:00:26.495462 12555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/master-0/wal/instance:
uuid: "5c88c5f91e97487c812ac51bc9cc7c10"
format_stamp: "Formatted at 2026-09-26 01:00:26 on dist-test-slave-7v10"
server_key: "40e4fa4eb417b4b39d003521b8d0ce18"
server_key_iv: "ef988b518736b49f8bf18c3e81387918"
server_key_version: "encryptionkey@0"
I20260926 01:00:26.496587 12555 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:00:26.497224 12569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:26.497390 12555 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:00:26.497479 12555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/master-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/master-0/wal
uuid: "5c88c5f91e97487c812ac51bc9cc7c10"
format_stamp: "Formatted at 2026-09-26 01:00:26 on dist-test-slave-7v10"
server_key: "40e4fa4eb417b4b39d003521b8d0ce18"
server_key_iv: "ef988b518736b49f8bf18c3e81387918"
server_key_version: "encryptionkey@0"
I20260926 01:00:26.497562 12555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:00:26.507732 12555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:00:26.507977 12555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:00:26.508116 12555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:00:26.511818 12555 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.62:33735
I20260926 01:00:26.511847 12621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.62:33735 every 8 connection(s)
I20260926 01:00:26.512141 12555 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
I20260926 01:00:26.512944 12622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:00:26.515079 12622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c88c5f91e97487c812ac51bc9cc7c10: Bootstrap starting.
I20260926 01:00:26.515592 12622 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c88c5f91e97487c812ac51bc9cc7c10: Neither blocks nor log segments found. Creating new log.
I20260926 01:00:26.515815 12622 log.cc:824] T 00000000000000000000000000000000 P 5c88c5f91e97487c812ac51bc9cc7c10: Log is configured to *not* fsync() on all Append() calls
I20260926 01:00:26.516137 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 12555
I20260926 01:00:26.516244 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/master-0/wal/instance
I20260926 01:00:26.516412 3372 external_mini_cluster.cc:1428] Setting key 6aced0649e3d9e99b72a1f0b92fae432
I20260926 01:00:26.516491 12622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c88c5f91e97487c812ac51bc9cc7c10: No bootstrap required, opened a new log
I20260926 01:00:26.518757 12622 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c88c5f91e97487c812ac51bc9cc7c10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c88c5f91e97487c812ac51bc9cc7c10" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 33735 } }
I20260926 01:00:26.518952 12622 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c88c5f91e97487c812ac51bc9cc7c10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:00:26.519006 12622 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c88c5f91e97487c812ac51bc9cc7c10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c88c5f91e97487c812ac51bc9cc7c10, State: Initialized, Role: FOLLOWER
I20260926 01:00:26.519142 12622 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c88c5f91e97487c812ac51bc9cc7c10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c88c5f91e97487c812ac51bc9cc7c10" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 33735 } }
I20260926 01:00:26.519248 12622 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c88c5f91e97487c812ac51bc9cc7c10 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:00:26.519284 12622 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c88c5f91e97487c812ac51bc9cc7c10 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:00:26.519339 12622 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c88c5f91e97487c812ac51bc9cc7c10 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:00:26.519984 12622 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c88c5f91e97487c812ac51bc9cc7c10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c88c5f91e97487c812ac51bc9cc7c10" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 33735 } }
I20260926 01:00:26.520087 12622 leader_election.cc:302] T 00000000000000000000000000000000 P 5c88c5f91e97487c812ac51bc9cc7c10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c88c5f91e97487c812ac51bc9cc7c10; no voters:
I20260926 01:00:26.520253 12622 leader_election.cc:288] T 00000000000000000000000000000000 P 5c88c5f91e97487c812ac51bc9cc7c10 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:00:26.520277 12627 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c88c5f91e97487c812ac51bc9cc7c10 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:00:26.520385 12627 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c88c5f91e97487c812ac51bc9cc7c10 [term 1 LEADER]: Becoming Leader. State: Replica: 5c88c5f91e97487c812ac51bc9cc7c10, State: Running, Role: LEADER
I20260926 01:00:26.520509 12622 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c88c5f91e97487c812ac51bc9cc7c10 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:00:26.520529 12627 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c88c5f91e97487c812ac51bc9cc7c10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c88c5f91e97487c812ac51bc9cc7c10" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 33735 } }
I20260926 01:00:26.520992 12629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c88c5f91e97487c812ac51bc9cc7c10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c88c5f91e97487c812ac51bc9cc7c10. Latest consensus state: current_term: 1 leader_uuid: "5c88c5f91e97487c812ac51bc9cc7c10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c88c5f91e97487c812ac51bc9cc7c10" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 33735 } } }
I20260926 01:00:26.521100 12629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c88c5f91e97487c812ac51bc9cc7c10 [sys.catalog]: This master's current role is: LEADER
I20260926 01:00:26.521510 12628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c88c5f91e97487c812ac51bc9cc7c10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c88c5f91e97487c812ac51bc9cc7c10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c88c5f91e97487c812ac51bc9cc7c10" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 33735 } } }
I20260926 01:00:26.521654 12628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c88c5f91e97487c812ac51bc9cc7c10 [sys.catalog]: This master's current role is: LEADER
I20260926 01:00:26.521835 12636 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:00:26.522414 12636 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:00:26.523674 12626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:26.516881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53358 (local address 127.3.75.62:33735)
0926 01:00:26.517129 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:26.517138 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:26.517174 (+ 36us) server_negotiation.cc:407] Connection header received
0926 01:00:26.517839 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:26.517848 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:26.517929 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:26.518087 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:26.518245 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:26.520075 (+ 1830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:26.521115 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:26.521725 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:26.521818 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:26.522801 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:26.522805 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:26.522805 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:00:26.523053 (+ 248us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:26.523098 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:00:26.523112 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:26.523173 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:26.523369 (+ 196us) server_negotiation.cc:299] Negotiation successful
0926 01:00:26.523447 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":108,"threads_started":1}
I20260926 01:00:26.524024 12636 catalog_manager.cc:1347] Generated new cluster ID: 6df6a32ec5674d46a6e00fe58180e698
I20260926 01:00:26.524129 12636 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:00:26.533337 12636 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:00:26.533829 12636 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:00:26.541999 12636 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c88c5f91e97487c812ac51bc9cc7c10: Generated new TSK 0
I20260926 01:00:26.542222 12636 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:00:26.544203 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.1:0
--local_ip_for_outbound_sockets=127.3.75.1
--webserver_interface=127.3.75.1
--webserver_port=0
--tserver_master_addrs=127.3.75.62:33735
--builtin_ntp_servers=127.3.75.20:34359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 01:00:26.640673 12646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:00:26.640857 12646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:00:26.640878 12646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:00:26.641997 12646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:00:26.642043 12646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:00:26.642068 12646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.1
I20260926 01:00:26.643409 12646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:34359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.75.1
--webserver_port=0
--tserver_master_addrs=127.3.75.62:33735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.1
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:00:26.643628 12646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:00:26.643916 12646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:00:26.649818 12651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:00:26.649819 12652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:00:26.649940 12654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:00:26.650101 12646 server_base.cc:1034] running on GCE node
I20260926 01:00:26.650313 12646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:00:26.650604 12646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:00:26.651793 12646 hybrid_clock.cc:648] HybridClock initialized: now 1790384426651778 us; error 28 us; skew 500 ppm
I20260926 01:00:26.653316 12646 webserver.cc:460] Webserver started at http://127.3.75.1:46373/ using document root <none> and password file <none>
I20260926 01:00:26.653589 12646 fs_manager.cc:359] Metadata directory not provided
I20260926 01:00:26.653658 12646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:00:26.653791 12646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:00:26.655252 12646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-0/data/instance:
uuid: "58e4b4d2457d4a5d8fce469ed867f244"
format_stamp: "Formatted at 2026-09-26 01:00:26 on dist-test-slave-7v10"
server_key: "dde7e366befd3a37efe3ffa64691af75"
server_key_iv: "f69ad7635dcfbac319403158763bf8fd"
server_key_version: "encryptionkey@0"
I20260926 01:00:26.655687 12646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-0/wal/instance:
uuid: "58e4b4d2457d4a5d8fce469ed867f244"
format_stamp: "Formatted at 2026-09-26 01:00:26 on dist-test-slave-7v10"
server_key: "dde7e366befd3a37efe3ffa64691af75"
server_key_iv: "f69ad7635dcfbac319403158763bf8fd"
server_key_version: "encryptionkey@0"
I20260926 01:00:26.657382 12646 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:00:26.658237 12660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:26.658406 12646 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:00:26.658512 12646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-0/wal
uuid: "58e4b4d2457d4a5d8fce469ed867f244"
format_stamp: "Formatted at 2026-09-26 01:00:26 on dist-test-slave-7v10"
server_key: "dde7e366befd3a37efe3ffa64691af75"
server_key_iv: "f69ad7635dcfbac319403158763bf8fd"
server_key_version: "encryptionkey@0"
I20260926 01:00:26.658582 12646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:00:26.673295 12646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:00:26.673605 12646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:00:26.673761 12646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:00:26.674000 12646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:00:26.674337 12646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:00:26.674382 12646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:26.674409 12646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:00:26.674455 12646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:26.680730 12646 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.1:41443
I20260926 01:00:26.680783 12773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.1:41443 every 8 connection(s)
I20260926 01:00:26.681131 12646 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
I20260926 01:00:26.686077 12626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:26.681878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:40895 (local address 127.3.75.62:33735)
0926 01:00:26.681936 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:26.681938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:26.682165 (+ 227us) server_negotiation.cc:407] Connection header received
0926 01:00:26.682634 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:26.682635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:26.682648 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:26.682718 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:26.683395 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:26.683896 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:26.684813 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:26.685035 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:26.685098 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:26.685825 (+ 727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:26.685826 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:26.685827 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:00:26.685831 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:26.685844 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:00:26.685848 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:26.685915 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:26.685997 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 01:00:26.686019 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 01:00:26.686239 12776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:26.681845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:33735 (local address 127.3.75.1:40895)
0926 01:00:26.682031 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:26.682072 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:26.682554 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:26.682788 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:26.682791 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:26.682799 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:26.683317 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:26.683321 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:26.683976 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:26.683977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:26.684733 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:26.684735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:26.685110 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:26.685110 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:26.685211 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:26.685697 (+ 486us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 01:00:26.685697 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:26.685724 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 01:00:26.685744 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:00:26.685922 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:00:26.685925 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:00:26.685928 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:00:26.685977 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 01:00:26.686052 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":88,"threads_started":1}
I20260926 01:00:26.686793 12774 heartbeater.cc:347] Connected to a master server at 127.3.75.62:33735
I20260926 01:00:26.686919 12774 heartbeater.cc:464] Registering TS with master...
I20260926 01:00:26.687095 12774 heartbeater.cc:511] Master 127.3.75.62:33735 requested a full tablet report, sending...
I20260926 01:00:26.687582 12586 ts_manager.cc:194] Registered new tserver with Master: 58e4b4d2457d4a5d8fce469ed867f244 (127.3.75.1:41443)
I20260926 01:00:26.688217 12586 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.75.1:40895
I20260926 01:00:26.689833 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 12646
I20260926 01:00:26.689916 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-0/wal/instance
I20260926 01:00:26.690141 3372 external_mini_cluster.cc:1428] Setting key f7cdc94c94d7101dc5c9d58c6cbb855f
I20260926 01:00:26.691049 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.2:0
--local_ip_for_outbound_sockets=127.3.75.2
--webserver_interface=127.3.75.2
--webserver_port=0
--tserver_master_addrs=127.3.75.62:33735
--builtin_ntp_servers=127.3.75.20:34359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 01:00:26.770797 12777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:00:26.771028 12777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:00:26.771065 12777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:00:26.772183 12777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:00:26.772245 12777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:00:26.772284 12777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.2
I20260926 01:00:26.773591 12777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:34359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.75.2
--webserver_port=0
--tserver_master_addrs=127.3.75.62:33735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.2
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:00:26.773842 12777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:00:26.774127 12777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:00:26.778924 12783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:00:26.779099 12777 server_base.cc:1034] running on GCE node
W20260926 01:00:26.778995 12785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:00:26.778945 12782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:00:26.779408 12777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:00:26.779670 12777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:00:26.780823 12777 hybrid_clock.cc:648] HybridClock initialized: now 1790384426780802 us; error 48 us; skew 500 ppm
I20260926 01:00:26.781971 12777 webserver.cc:460] Webserver started at http://127.3.75.2:40363/ using document root <none> and password file <none>
I20260926 01:00:26.782224 12777 fs_manager.cc:359] Metadata directory not provided
I20260926 01:00:26.782291 12777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:00:26.782440 12777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:00:26.783506 12777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-1/data/instance:
uuid: "58937329fb0040699771c0ddf5ff0802"
format_stamp: "Formatted at 2026-09-26 01:00:26 on dist-test-slave-7v10"
server_key: "c67b51713599d9313b0d7acbf123b25d"
server_key_iv: "16b4cf354fcaa2906806c8ffed5f703d"
server_key_version: "encryptionkey@0"
I20260926 01:00:26.783869 12777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-1/wal/instance:
uuid: "58937329fb0040699771c0ddf5ff0802"
format_stamp: "Formatted at 2026-09-26 01:00:26 on dist-test-slave-7v10"
server_key: "c67b51713599d9313b0d7acbf123b25d"
server_key_iv: "16b4cf354fcaa2906806c8ffed5f703d"
server_key_version: "encryptionkey@0"
I20260926 01:00:26.785144 12777 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:00:26.785809 12791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:26.785969 12777 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:00:26.786064 12777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-1/wal
uuid: "58937329fb0040699771c0ddf5ff0802"
format_stamp: "Formatted at 2026-09-26 01:00:26 on dist-test-slave-7v10"
server_key: "c67b51713599d9313b0d7acbf123b25d"
server_key_iv: "16b4cf354fcaa2906806c8ffed5f703d"
server_key_version: "encryptionkey@0"
I20260926 01:00:26.786140 12777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:00:26.816905 12777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:00:26.817239 12777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:00:26.817396 12777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:00:26.817633 12777 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:00:26.817979 12777 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:00:26.818032 12777 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:26.818065 12777 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:00:26.818094 12777 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:26.824155 12777 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.2:35711
I20260926 01:00:26.824204 12904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.2:35711 every 8 connection(s)
I20260926 01:00:26.824522 12777 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
I20260926 01:00:26.825671 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 12777
I20260926 01:00:26.825745 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-1/wal/instance
I20260926 01:00:26.825922 3372 external_mini_cluster.cc:1428] Setting key ec517b5b1fb3f31b112750e1db099877
I20260926 01:00:26.826982 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.3:0
--local_ip_for_outbound_sockets=127.3.75.3
--webserver_interface=127.3.75.3
--webserver_port=0
--tserver_master_addrs=127.3.75.62:33735
--builtin_ntp_servers=127.3.75.20:34359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 01:00:26.830621 12626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:26.825236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:55729 (local address 127.3.75.62:33735)
0926 01:00:26.825292 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:26.825294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:26.825518 (+ 224us) server_negotiation.cc:407] Connection header received
0926 01:00:26.826296 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:26.826297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:26.826311 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:26.826362 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:26.827455 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:26.828001 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:26.829011 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:26.829338 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:26.829417 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:26.830266 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:26.830268 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:26.830268 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:00:26.830274 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:26.830293 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:00:26.830299 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:26.830367 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:26.830459 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 01:00:26.830497 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 01:00:26.830668 12907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:26.825229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:33735 (local address 127.3.75.2:55729)
0926 01:00:26.825402 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:26.825435 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:26.826114 (+ 679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:26.826642 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:26.826646 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:26.826657 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:26.827347 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:26.827353 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:26.828106 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:26.828107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:26.828910 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:26.828913 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:26.829439 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:26.829441 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:26.829558 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:26.830109 (+ 551us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 01:00:26.830109 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:26.830136 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 01:00:26.830156 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:00:26.830376 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:00:26.830381 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:00:26.830383 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 01:00:26.830439 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 01:00:26.830518 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":75,"threads_started":1}
I20260926 01:00:26.831185 12905 heartbeater.cc:347] Connected to a master server at 127.3.75.62:33735
I20260926 01:00:26.831271 12905 heartbeater.cc:464] Registering TS with master...
I20260926 01:00:26.831449 12905 heartbeater.cc:511] Master 127.3.75.62:33735 requested a full tablet report, sending...
I20260926 01:00:26.831796 12586 ts_manager.cc:194] Registered new tserver with Master: 58937329fb0040699771c0ddf5ff0802 (127.3.75.2:35711)
I20260926 01:00:26.832129 12586 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.75.2:55729
W20260926 01:00:26.909384 12908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:00:26.909619 12908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:00:26.909652 12908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:00:26.910810 12908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:00:26.910904 12908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:00:26.910945 12908 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.3
I20260926 01:00:26.912272 12908 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:34359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.75.3
--webserver_port=0
--tserver_master_addrs=127.3.75.62:33735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.3
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:00:26.912529 12908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:00:26.912822 12908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:00:26.917842 12913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:00:26.917846 12914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:00:26.917846 12916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:00:26.918171 12908 server_base.cc:1034] running on GCE node
I20260926 01:00:26.918401 12908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:00:26.918642 12908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:00:26.919800 12908 hybrid_clock.cc:648] HybridClock initialized: now 1790384426919781 us; error 29 us; skew 500 ppm
I20260926 01:00:26.921181 12908 webserver.cc:460] Webserver started at http://127.3.75.3:45773/ using document root <none> and password file <none>
I20260926 01:00:26.921438 12908 fs_manager.cc:359] Metadata directory not provided
I20260926 01:00:26.921505 12908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:00:26.921654 12908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:00:26.922720 12908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-2/data/instance:
uuid: "280becf12f324bb99a92236c631168df"
format_stamp: "Formatted at 2026-09-26 01:00:26 on dist-test-slave-7v10"
server_key: "41678560328ac41ff1012a335a7a3a90"
server_key_iv: "2c355d20ecee91179dae21d353e20f53"
server_key_version: "encryptionkey@0"
I20260926 01:00:26.923092 12908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-2/wal/instance:
uuid: "280becf12f324bb99a92236c631168df"
format_stamp: "Formatted at 2026-09-26 01:00:26 on dist-test-slave-7v10"
server_key: "41678560328ac41ff1012a335a7a3a90"
server_key_iv: "2c355d20ecee91179dae21d353e20f53"
server_key_version: "encryptionkey@0"
I20260926 01:00:26.924316 12908 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:00:26.924955 12922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:26.925128 12908 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 01:00:26.925196 12908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-2/wal
uuid: "280becf12f324bb99a92236c631168df"
format_stamp: "Formatted at 2026-09-26 01:00:26 on dist-test-slave-7v10"
server_key: "41678560328ac41ff1012a335a7a3a90"
server_key_iv: "2c355d20ecee91179dae21d353e20f53"
server_key_version: "encryptionkey@0"
I20260926 01:00:26.925246 12908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:00:26.951201 12908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:00:26.951490 12908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:00:26.951596 12908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:00:26.951778 12908 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:00:26.952167 12908 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:00:26.952219 12908 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:26.952263 12908 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:00:26.952292 12908 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:26.957844 12908 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.3:45267
I20260926 01:00:26.957890 13035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.3:45267 every 8 connection(s)
I20260926 01:00:26.958240 12908 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
I20260926 01:00:26.962359 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 12908
I20260926 01:00:26.962455 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-2/wal/instance
I20260926 01:00:26.962646 3372 external_mini_cluster.cc:1428] Setting key 6b4daf4a18a0ee35db2b0019705010ba
I20260926 01:00:26.963600 12626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:26.959010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.3:52879 (local address 127.3.75.62:33735)
0926 01:00:26.959069 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:26.959071 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:26.959316 (+ 245us) server_negotiation.cc:407] Connection header received
0926 01:00:26.959809 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:26.959810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:26.959825 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:26.959902 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:26.960531 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:26.961026 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:26.962049 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:26.962291 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:26.962368 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:26.963315 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:26.963316 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:26.963316 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:00:26.963320 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:26.963340 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:00:26.963344 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:26.963388 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:26.963492 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 01:00:26.963517 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 01:00:26.963832 13038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:26.959046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:33735 (local address 127.3.75.3:52879)
0926 01:00:26.959197 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:26.959231 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:26.959711 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:26.959954 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:26.959957 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:26.959966 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:26.960463 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:26.960467 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:26.961099 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:26.961100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:26.961977 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:26.961979 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:26.962379 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:26.962380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:26.962519 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:26.963166 (+ 647us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 01:00:26.963167 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:26.963205 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 01:00:26.963233 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:00:26.963404 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:00:26.963409 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:00:26.963412 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:00:26.963493 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 01:00:26.963609 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":64,"threads_started":1}
I20260926 01:00:26.964323 13036 heartbeater.cc:347] Connected to a master server at 127.3.75.62:33735
I20260926 01:00:26.964419 13036 heartbeater.cc:464] Registering TS with master...
I20260926 01:00:26.964602 13036 heartbeater.cc:511] Master 127.3.75.62:33735 requested a full tablet report, sending...
I20260926 01:00:26.964927 12586 ts_manager.cc:194] Registered new tserver with Master: 280becf12f324bb99a92236c631168df (127.3.75.3:45267)
I20260926 01:00:26.965255 12586 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.75.3:52879
I20260926 01:00:26.974350 3372 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 01:00:26.976120 12626 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:00:26.975731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53374 (local address 127.3.75.62:33735)
0926 01:00:26.975782 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:26.975784 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:26.975870 (+ 86us) server_negotiation.cc:407] Connection header received
0926 01:00:26.975909 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:26.975909 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:26.975917 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:26.975960 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:26.976070 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53374: BlockingRecv error: recv got EOF from 127.0.0.1:53374 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 01:00:26.976192 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 12646
I20260926 01:00:26.981000 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 12777
I20260926 01:00:26.985033 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 12908
I20260926 01:00:26.989393 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 12555
2026-09-26T01:00:26Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (613 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-26T01:00:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:00:27Z Disabled control of system clock
I20260926 01:00:27.015272 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:39637
--webserver_interface=127.3.75.62
--webserver_port=0
--builtin_ntp_servers=127.3.75.20:45633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.75.62:39637
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 01:00:27.093509 13047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:00:27.093694 13047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:00:27.093713 13047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:00:27.094759 13047 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:00:27.094792 13047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:00:27.094805 13047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:00:27.094817 13047 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:00:27.094830 13047 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:00:27.096180 13047 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:45633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.75.62:39637
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:39637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:00:27.096359 13047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:00:27.096565 13047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:00:27.100987 13053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:00:27.101043 13055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:00:27.100998 13052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:00:27.101313 13047 server_base.cc:1034] running on GCE node
I20260926 01:00:27.101512 13047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:00:27.101733 13047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:00:27.102907 13047 hybrid_clock.cc:648] HybridClock initialized: now 1790384427102897 us; error 33 us; skew 500 ppm
I20260926 01:00:27.104063 13047 webserver.cc:460] Webserver started at http://127.3.75.62:36677/ using document root <none> and password file <none>
I20260926 01:00:27.104280 13047 fs_manager.cc:359] Metadata directory not provided
I20260926 01:00:27.104323 13047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:00:27.104462 13047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:00:27.105453 13047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/master-0/data/instance:
uuid: "575e065950e7451fb6baac5e2e2243b1"
format_stamp: "Formatted at 2026-09-26 01:00:27 on dist-test-slave-7v10"
server_key: "f393d5ca5ddcba0c86deca36bb1ff9cf"
server_key_iv: "bde5d155e0c1e95897d18a0579956f2b"
server_key_version: "encryptionkey@0"
I20260926 01:00:27.105793 13047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/master-0/wal/instance:
uuid: "575e065950e7451fb6baac5e2e2243b1"
format_stamp: "Formatted at 2026-09-26 01:00:27 on dist-test-slave-7v10"
server_key: "f393d5ca5ddcba0c86deca36bb1ff9cf"
server_key_iv: "bde5d155e0c1e95897d18a0579956f2b"
server_key_version: "encryptionkey@0"
I20260926 01:00:27.107064 13047 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:00:27.107782 13061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:27.107957 13047 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:00:27.108042 13047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/master-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/master-0/wal
uuid: "575e065950e7451fb6baac5e2e2243b1"
format_stamp: "Formatted at 2026-09-26 01:00:27 on dist-test-slave-7v10"
server_key: "f393d5ca5ddcba0c86deca36bb1ff9cf"
server_key_iv: "bde5d155e0c1e95897d18a0579956f2b"
server_key_version: "encryptionkey@0"
I20260926 01:00:27.108109 13047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:00:27.111289 13047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:00:27.111505 13047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:00:27.111656 13047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:00:27.115200 13047 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.62:39637
I20260926 01:00:27.115257 13113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.62:39637 every 8 connection(s)
I20260926 01:00:27.115547 13047 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
I20260926 01:00:27.116124 13114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:00:27.118633 13114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 575e065950e7451fb6baac5e2e2243b1: Bootstrap starting.
I20260926 01:00:27.119104 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 13047
I20260926 01:00:27.119199 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/master-0/wal/instance
I20260926 01:00:27.119323 13114 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 575e065950e7451fb6baac5e2e2243b1: Neither blocks nor log segments found. Creating new log.
I20260926 01:00:27.119385 3372 external_mini_cluster.cc:1428] Setting key d9b9ffe077f69026acf4e01c9135d3e5
I20260926 01:00:27.119652 13114 log.cc:824] T 00000000000000000000000000000000 P 575e065950e7451fb6baac5e2e2243b1: Log is configured to *not* fsync() on all Append() calls
I20260926 01:00:27.120447 13114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 575e065950e7451fb6baac5e2e2243b1: No bootstrap required, opened a new log
I20260926 01:00:27.121842 13118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:27.119836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40054 (local address 127.3.75.62:39637)
0926 01:00:27.119989 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:27.120001 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:27.120024 (+ 23us) server_negotiation.cc:407] Connection header received
0926 01:00:27.120572 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:27.120583 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:27.120643 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:27.120789 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:27.121399 (+ 610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:27.121403 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:27.121403 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:00:27.121408 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:27.121453 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:00:27.121466 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:27.121565 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:27.121601 (+ 36us) server_negotiation.cc:299] Negotiation successful
0926 01:00:27.121654 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":59,"threads_started":1}
I20260926 01:00:27.122400 13114 raft_consensus.cc:348] T 00000000000000000000000000000000 P 575e065950e7451fb6baac5e2e2243b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "575e065950e7451fb6baac5e2e2243b1" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 39637 } }
I20260926 01:00:27.122563 13114 raft_consensus.cc:374] T 00000000000000000000000000000000 P 575e065950e7451fb6baac5e2e2243b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:00:27.122609 13114 raft_consensus.cc:729] T 00000000000000000000000000000000 P 575e065950e7451fb6baac5e2e2243b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 575e065950e7451fb6baac5e2e2243b1, State: Initialized, Role: FOLLOWER
I20260926 01:00:27.122725 13114 consensus_queue.cc:260] T 00000000000000000000000000000000 P 575e065950e7451fb6baac5e2e2243b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "575e065950e7451fb6baac5e2e2243b1" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 39637 } }
I20260926 01:00:27.122808 13114 raft_consensus.cc:388] T 00000000000000000000000000000000 P 575e065950e7451fb6baac5e2e2243b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:00:27.122852 13114 raft_consensus.cc:482] T 00000000000000000000000000000000 P 575e065950e7451fb6baac5e2e2243b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:00:27.122921 13114 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 575e065950e7451fb6baac5e2e2243b1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:00:27.123714 13114 raft_consensus.cc:504] T 00000000000000000000000000000000 P 575e065950e7451fb6baac5e2e2243b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "575e065950e7451fb6baac5e2e2243b1" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 39637 } }
I20260926 01:00:27.123834 13114 leader_election.cc:302] T 00000000000000000000000000000000 P 575e065950e7451fb6baac5e2e2243b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 575e065950e7451fb6baac5e2e2243b1; no voters:
I20260926 01:00:27.124003 13114 leader_election.cc:288] T 00000000000000000000000000000000 P 575e065950e7451fb6baac5e2e2243b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:00:27.124022 13119 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 575e065950e7451fb6baac5e2e2243b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:00:27.124154 13119 raft_consensus.cc:686] T 00000000000000000000000000000000 P 575e065950e7451fb6baac5e2e2243b1 [term 1 LEADER]: Becoming Leader. State: Replica: 575e065950e7451fb6baac5e2e2243b1, State: Running, Role: LEADER
I20260926 01:00:27.124253 13114 sys_catalog.cc:564] T 00000000000000000000000000000000 P 575e065950e7451fb6baac5e2e2243b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:00:27.124300 13119 consensus_queue.cc:237] T 00000000000000000000000000000000 P 575e065950e7451fb6baac5e2e2243b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "575e065950e7451fb6baac5e2e2243b1" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 39637 } }
I20260926 01:00:27.124693 13120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 575e065950e7451fb6baac5e2e2243b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "575e065950e7451fb6baac5e2e2243b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "575e065950e7451fb6baac5e2e2243b1" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 39637 } } }
I20260926 01:00:27.124788 13120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 575e065950e7451fb6baac5e2e2243b1 [sys.catalog]: This master's current role is: LEADER
I20260926 01:00:27.125002 13127 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:00:27.125514 13127 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:00:27.124967 13121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 575e065950e7451fb6baac5e2e2243b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 575e065950e7451fb6baac5e2e2243b1. Latest consensus state: current_term: 1 leader_uuid: "575e065950e7451fb6baac5e2e2243b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "575e065950e7451fb6baac5e2e2243b1" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 39637 } } }
I20260926 01:00:27.126096 13121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 575e065950e7451fb6baac5e2e2243b1 [sys.catalog]: This master's current role is: LEADER
I20260926 01:00:27.127151 13127 catalog_manager.cc:1347] Generated new cluster ID: 7ab87ba8b76145f8b8212701095d9be8
I20260926 01:00:27.127214 13127 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:00:27.150852 13127 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:00:27.151086 13127 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:00:27.155710 13127 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 575e065950e7451fb6baac5e2e2243b1: Generated new TSK 0
I20260926 01:00:27.155920 13127 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:00:27.165213 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.1:0
--local_ip_for_outbound_sockets=127.3.75.1
--webserver_interface=127.3.75.1
--webserver_port=0
--tserver_master_addrs=127.3.75.62:39637
--builtin_ntp_servers=127.3.75.20:45633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 01:00:27.261045 13138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:00:27.261263 13138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:00:27.261298 13138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:00:27.262465 13138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:00:27.262519 13138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:00:27.262562 13138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.1
I20260926 01:00:27.263870 13138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:45633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.75.1
--webserver_port=0
--tserver_master_addrs=127.3.75.62:39637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.1
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:00:27.264147 13138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:00:27.264416 13138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:00:27.269312 13143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:00:27.269320 13146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:00:27.269380 13138 server_base.cc:1034] running on GCE node
W20260926 01:00:27.269325 13144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:00:27.269729 13138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:00:27.269917 13138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:00:27.271044 13138 hybrid_clock.cc:648] HybridClock initialized: now 1790384427271034 us; error 25 us; skew 500 ppm
I20260926 01:00:27.272176 13138 webserver.cc:460] Webserver started at http://127.3.75.1:36713/ using document root <none> and password file <none>
I20260926 01:00:27.272396 13138 fs_manager.cc:359] Metadata directory not provided
I20260926 01:00:27.272440 13138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:00:27.272578 13138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:00:27.273597 13138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-0/data/instance:
uuid: "07f985cedb964ef08062bace589f2957"
format_stamp: "Formatted at 2026-09-26 01:00:27 on dist-test-slave-7v10"
server_key: "b73decb7c5ecb5bd40581a0f99b21e90"
server_key_iv: "f5c66cf3dff41f06a78fe0827ce05d93"
server_key_version: "encryptionkey@0"
I20260926 01:00:27.273944 13138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-0/wal/instance:
uuid: "07f985cedb964ef08062bace589f2957"
format_stamp: "Formatted at 2026-09-26 01:00:27 on dist-test-slave-7v10"
server_key: "b73decb7c5ecb5bd40581a0f99b21e90"
server_key_iv: "f5c66cf3dff41f06a78fe0827ce05d93"
server_key_version: "encryptionkey@0"
I20260926 01:00:27.275224 13138 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:00:27.275890 13152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:27.276048 13138 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:00:27.276144 13138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-0/wal
uuid: "07f985cedb964ef08062bace589f2957"
format_stamp: "Formatted at 2026-09-26 01:00:27 on dist-test-slave-7v10"
server_key: "b73decb7c5ecb5bd40581a0f99b21e90"
server_key_iv: "f5c66cf3dff41f06a78fe0827ce05d93"
server_key_version: "encryptionkey@0"
I20260926 01:00:27.276213 13138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:00:27.279541 13138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:00:27.279789 13138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:00:27.279927 13138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:00:27.280136 13138 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:00:27.280428 13138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:00:27.280478 13138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:27.280510 13138 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:00:27.280536 13138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:27.286115 13138 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.1:38373
I20260926 01:00:27.286157 13265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.1:38373 every 8 connection(s)
I20260926 01:00:27.286459 13138 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
I20260926 01:00:27.289300 13118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:27.287116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:60795 (local address 127.3.75.62:39637)
0926 01:00:27.287159 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:27.287161 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:27.287380 (+ 219us) server_negotiation.cc:407] Connection header received
0926 01:00:27.287871 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:27.287872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:27.287879 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:27.287936 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:27.289023 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:27.289025 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:27.289025 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:00:27.289030 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:27.289062 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:00:27.289066 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:27.289136 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:27.289215 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 01:00:27.289230 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 01:00:27.289487 13268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:27.287098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:39637 (local address 127.3.75.1:60795)
0926 01:00:27.287259 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:27.287290 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:27.287770 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:27.288007 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:27.288011 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:27.288021 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:27.288877 (+ 856us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 01:00:27.288878 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:27.288904 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 01:00:27.288931 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:00:27.289135 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:00:27.289142 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:00:27.289145 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:00:27.289225 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 01:00:27.289284 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":71,"threads_started":1}
I20260926 01:00:27.289950 13266 heartbeater.cc:347] Connected to a master server at 127.3.75.62:39637
I20260926 01:00:27.290045 13266 heartbeater.cc:464] Registering TS with master...
I20260926 01:00:27.290155 13266 heartbeater.cc:511] Master 127.3.75.62:39637 requested a full tablet report, sending...
I20260926 01:00:27.290544 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 13138
I20260926 01:00:27.290649 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-0/wal/instance
I20260926 01:00:27.290670 13078 ts_manager.cc:194] Registered new tserver with Master: 07f985cedb964ef08062bace589f2957 (127.3.75.1:38373)
I20260926 01:00:27.290823 3372 external_mini_cluster.cc:1428] Setting key 9d17c69defc69f976a723025b39834ba
I20260926 01:00:27.291675 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.2:0
--local_ip_for_outbound_sockets=127.3.75.2
--webserver_interface=127.3.75.2
--webserver_port=0
--tserver_master_addrs=127.3.75.62:39637
--builtin_ntp_servers=127.3.75.20:45633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 01:00:27.369957 13269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:00:27.370189 13269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:00:27.370226 13269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:00:27.371381 13269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:00:27.371450 13269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:00:27.371505 13269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.2
I20260926 01:00:27.372828 13269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:45633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.75.2
--webserver_port=0
--tserver_master_addrs=127.3.75.62:39637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13269
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.2
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:00:27.373078 13269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:00:27.373350 13269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:00:27.378285 13277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:00:27.378345 13275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:00:27.378419 13269 server_base.cc:1034] running on GCE node
W20260926 01:00:27.378343 13274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:00:27.378772 13269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:00:27.379067 13269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:00:27.380209 13269 hybrid_clock.cc:648] HybridClock initialized: now 1790384427380192 us; error 36 us; skew 500 ppm
I20260926 01:00:27.381319 13269 webserver.cc:460] Webserver started at http://127.3.75.2:41485/ using document root <none> and password file <none>
I20260926 01:00:27.381551 13269 fs_manager.cc:359] Metadata directory not provided
I20260926 01:00:27.381595 13269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:00:27.381742 13269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:00:27.382750 13269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-1/data/instance:
uuid: "967159023f094a559da9e66c740734ba"
format_stamp: "Formatted at 2026-09-26 01:00:27 on dist-test-slave-7v10"
server_key: "5e57d22356f6f50017fed8641c166e5a"
server_key_iv: "35dce57e3f058177a3565201255bf6fc"
server_key_version: "encryptionkey@0"
I20260926 01:00:27.383131 13269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-1/wal/instance:
uuid: "967159023f094a559da9e66c740734ba"
format_stamp: "Formatted at 2026-09-26 01:00:27 on dist-test-slave-7v10"
server_key: "5e57d22356f6f50017fed8641c166e5a"
server_key_iv: "35dce57e3f058177a3565201255bf6fc"
server_key_version: "encryptionkey@0"
I20260926 01:00:27.384374 13269 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:00:27.385033 13283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:27.385193 13269 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:00:27.385275 13269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-1/wal
uuid: "967159023f094a559da9e66c740734ba"
format_stamp: "Formatted at 2026-09-26 01:00:27 on dist-test-slave-7v10"
server_key: "5e57d22356f6f50017fed8641c166e5a"
server_key_iv: "35dce57e3f058177a3565201255bf6fc"
server_key_version: "encryptionkey@0"
I20260926 01:00:27.385344 13269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:00:27.388607 13269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:00:27.388841 13269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:00:27.388972 13269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:00:27.389179 13269 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:00:27.389487 13269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:00:27.389521 13269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:27.389572 13269 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:00:27.389598 13269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:27.395109 13269 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.2:46141
I20260926 01:00:27.395152 13396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.2:46141 every 8 connection(s)
I20260926 01:00:27.395429 13269 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
I20260926 01:00:27.395854 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 13269
I20260926 01:00:27.395961 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-1/wal/instance
I20260926 01:00:27.396185 3372 external_mini_cluster.cc:1428] Setting key 747df8097cdcdf2a3dd4f24e363c4470
I20260926 01:00:27.397029 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.3:0
--local_ip_for_outbound_sockets=127.3.75.3
--webserver_interface=127.3.75.3
--webserver_port=0
--tserver_master_addrs=127.3.75.62:39637
--builtin_ntp_servers=127.3.75.20:45633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 01:00:27.398428 13118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:27.396046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:54921 (local address 127.3.75.62:39637)
0926 01:00:27.396116 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:27.396119 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:27.396348 (+ 229us) server_negotiation.cc:407] Connection header received
0926 01:00:27.396932 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:27.396934 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:27.396940 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:27.396985 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:27.398230 (+ 1245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:27.398232 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:27.398232 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:00:27.398236 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:27.398251 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:00:27.398255 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:27.398295 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:27.398367 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 01:00:27.398378 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 01:00:27.398730 13399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:27.396054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:39637 (local address 127.3.75.2:54921)
0926 01:00:27.396228 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:27.396263 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:27.396826 (+ 563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:27.397067 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:27.397072 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:27.397081 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:27.398065 (+ 984us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 01:00:27.398065 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:27.398104 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 01:00:27.398142 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:00:27.398296 (+ 154us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:00:27.398299 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:00:27.398302 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:00:27.398376 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 01:00:27.398438 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":91,"threads_started":1}
I20260926 01:00:27.399251 13397 heartbeater.cc:347] Connected to a master server at 127.3.75.62:39637
I20260926 01:00:27.399334 13397 heartbeater.cc:464] Registering TS with master...
I20260926 01:00:27.399466 13397 heartbeater.cc:511] Master 127.3.75.62:39637 requested a full tablet report, sending...
I20260926 01:00:27.399828 13078 ts_manager.cc:194] Registered new tserver with Master: 967159023f094a559da9e66c740734ba (127.3.75.2:46141)
W20260926 01:00:27.476720 13400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:00:27.476914 13400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:00:27.476934 13400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:00:27.478066 13400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:00:27.478104 13400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:00:27.478143 13400 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.3
I20260926 01:00:27.479578 13400 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:45633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.75.3
--webserver_port=0
--tserver_master_addrs=127.3.75.62:39637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.3
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:00:27.479797 13400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:00:27.480072 13400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:00:27.484803 13405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:00:27.484787 13408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:00:27.484812 13406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:00:27.485059 13400 server_base.cc:1034] running on GCE node
I20260926 01:00:27.485253 13400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:00:27.485474 13400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:00:27.486622 13400 hybrid_clock.cc:648] HybridClock initialized: now 1790384427486614 us; error 37 us; skew 500 ppm
I20260926 01:00:27.487879 13400 webserver.cc:460] Webserver started at http://127.3.75.3:36219/ using document root <none> and password file <none>
I20260926 01:00:27.488101 13400 fs_manager.cc:359] Metadata directory not provided
I20260926 01:00:27.488158 13400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:00:27.488276 13400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:00:27.489297 13400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-2/data/instance:
uuid: "e8db660465124e01b091449b417976e7"
format_stamp: "Formatted at 2026-09-26 01:00:27 on dist-test-slave-7v10"
server_key: "14c4fe923800ed7ec53ab6c4cb0e0eaf"
server_key_iv: "c2b1a280a48cce205d697ba7abc8c65a"
server_key_version: "encryptionkey@0"
I20260926 01:00:27.489641 13400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-2/wal/instance:
uuid: "e8db660465124e01b091449b417976e7"
format_stamp: "Formatted at 2026-09-26 01:00:27 on dist-test-slave-7v10"
server_key: "14c4fe923800ed7ec53ab6c4cb0e0eaf"
server_key_iv: "c2b1a280a48cce205d697ba7abc8c65a"
server_key_version: "encryptionkey@0"
I20260926 01:00:27.490948 13400 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:00:27.491664 13414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:27.491825 13400 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:00:27.491909 13400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-2/wal
uuid: "e8db660465124e01b091449b417976e7"
format_stamp: "Formatted at 2026-09-26 01:00:27 on dist-test-slave-7v10"
server_key: "14c4fe923800ed7ec53ab6c4cb0e0eaf"
server_key_iv: "c2b1a280a48cce205d697ba7abc8c65a"
server_key_version: "encryptionkey@0"
I20260926 01:00:27.491987 13400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:00:27.495173 13400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:00:27.495432 13400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:00:27.495571 13400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:00:27.495776 13400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:00:27.496043 13400 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:00:27.496089 13400 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:27.496126 13400 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:00:27.496153 13400 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:27.501438 13400 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.3:46543
I20260926 01:00:27.501485 13527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.3:46543 every 8 connection(s)
I20260926 01:00:27.501788 13400 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
I20260926 01:00:27.504422 13118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:27.502456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.3:34337 (local address 127.3.75.62:39637)
0926 01:00:27.502530 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:27.502532 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:27.502691 (+ 159us) server_negotiation.cc:407] Connection header received
0926 01:00:27.503149 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:27.503150 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:27.503155 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:27.503204 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:27.504255 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:27.504256 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:27.504256 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:00:27.504259 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:27.504268 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:00:27.504271 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:27.504306 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:27.504361 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 01:00:27.504371 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 01:00:27.504583 13530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:27.502423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:39637 (local address 127.3.75.3:34337)
0926 01:00:27.502592 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:27.502620 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:27.503071 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:27.503251 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:27.503253 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:27.503261 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:27.504141 (+ 880us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 01:00:27.504141 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:27.504166 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 01:00:27.504192 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:00:27.504313 (+ 121us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:00:27.504315 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:00:27.504317 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 01:00:27.504360 (+ 43us) client_negotiation.cc:240] Negotiation successful
0926 01:00:27.504402 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":93,"threads_started":1}
I20260926 01:00:27.505066 13528 heartbeater.cc:347] Connected to a master server at 127.3.75.62:39637
I20260926 01:00:27.505173 13528 heartbeater.cc:464] Registering TS with master...
I20260926 01:00:27.505302 13528 heartbeater.cc:511] Master 127.3.75.62:39637 requested a full tablet report, sending...
I20260926 01:00:27.505616 13078 ts_manager.cc:194] Registered new tserver with Master: e8db660465124e01b091449b417976e7 (127.3.75.3:46543)
I20260926 01:00:27.511858 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 13400
I20260926 01:00:27.511957 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384372073861-3372-0/minicluster-data/ts-2/wal/instance
I20260926 01:00:27.512146 3372 external_mini_cluster.cc:1428] Setting key 3eeed4b8122ac754ef109ceee1242485
I20260926 01:00:27.512917 3372 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 01:00:27.514608 13118 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:00:27.514098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40068 (local address 127.3.75.62:39637)
0926 01:00:27.514170 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:27.514172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:27.514259 (+ 87us) server_negotiation.cc:407] Connection header received
0926 01:00:27.514347 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:27.514348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:27.514353 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:27.514415 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:27.514542 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40068: BlockingRecv error: recv got EOF from 127.0.0.1:40068 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 01:00:27.514683 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 13138
I20260926 01:00:27.519304 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 13269
I20260926 01:00:27.524000 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 13400
I20260926 01:00:27.528329 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 13047
2026-09-26T01:00:27Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (539 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:00:27 dist-test-slave-7v10 krb5kdc[13538](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:00:27 dist-test-slave-7v10 krb5kdc[13538](info): set up 2 sockets
Sep 26 01:00:27 dist-test-slave-7v10 krb5kdc[13538](info): commencing operation
krb5kdc: starting...
W20260926 01:00:28.578846 3372 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Sep 26 01:00:28 dist-test-slave-7v10 krb5kdc[13538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:00:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:00:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.75.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab.
Entry for principal kudu/127.3.75.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab.
I20260926 01:00:28.697810 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:41719
--webserver_interface=127.3.75.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:36085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.75.62:41719
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:00:28.778918 13551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:00:28.779119 13551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:00:28.779140 13551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:00:28.780279 13551 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:00:28.780320 13551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:00:28.780335 13551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:00:28.780369 13551 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:00:28.780397 13551 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:00:28.781972 13551 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:36085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.75.62:41719
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:41719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:00:28.782212 13551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:00:28.782425 13551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:00:28.787120 13557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:00:28.787114 13556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:00:28.787231 13551 server_base.cc:1034] running on GCE node
W20260926 01:00:28.787190 13559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:00:28.787575 13551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:00:28.787830 13551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:00:28.788980 13551 hybrid_clock.cc:648] HybridClock initialized: now 1790384428788968 us; error 37 us; skew 500 ppm
Sep 26 01:00:28 dist-test-slave-7v10 krb5kdc[13538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384428, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:00:28.791321 13551 init.cc:377] Logged in from keytab as kudu/127.3.75.62@KRBTEST.COM (short username kudu)
I20260926 01:00:28.791738 13551 webserver.cc:460] Webserver started at http://127.3.75.62:39931/ using document root <none> and password file <none>
I20260926 01:00:28.791952 13551 fs_manager.cc:359] Metadata directory not provided
I20260926 01:00:28.792017 13551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:00:28.792155 13551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:00:28.793138 13551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/master-0/data/instance:
uuid: "e8ef40897a7148b89e43cb5d562dee9a"
format_stamp: "Formatted at 2026-09-26 01:00:28 on dist-test-slave-7v10"
server_key: "5c8b8954734c7a4f48edd46b5b25b571"
server_key_iv: "6445a8769df72966f38c11b58d728ba6"
server_key_version: "encryptionkey@0"
I20260926 01:00:28.793465 13551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/master-0/wal/instance:
uuid: "e8ef40897a7148b89e43cb5d562dee9a"
format_stamp: "Formatted at 2026-09-26 01:00:28 on dist-test-slave-7v10"
server_key: "5c8b8954734c7a4f48edd46b5b25b571"
server_key_iv: "6445a8769df72966f38c11b58d728ba6"
server_key_version: "encryptionkey@0"
I20260926 01:00:28.794768 13551 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:00:28.795532 13566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:28.795717 13551 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 01:00:28.795810 13551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/master-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/master-0/wal
uuid: "e8ef40897a7148b89e43cb5d562dee9a"
format_stamp: "Formatted at 2026-09-26 01:00:28 on dist-test-slave-7v10"
server_key: "5c8b8954734c7a4f48edd46b5b25b571"
server_key_iv: "6445a8769df72966f38c11b58d728ba6"
server_key_version: "encryptionkey@0"
I20260926 01:00:28.795907 13551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:00:28.804747 13551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:00:28.806497 13551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:00:28.806650 13551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:00:28.810427 13551 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.62:41719
I20260926 01:00:28.810469 13618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.62:41719 every 8 connection(s)
I20260926 01:00:28.810817 13551 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
I20260926 01:00:28.811740 13619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:00:28.812247 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 13551
I20260926 01:00:28.812343 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/master-0/wal/instance
I20260926 01:00:28.812512 3372 external_mini_cluster.cc:1428] Setting key 76a1a37e5966506562c7fe41710f9f5b
I20260926 01:00:28.814800 13619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8ef40897a7148b89e43cb5d562dee9a: Bootstrap starting.
I20260926 01:00:28.815426 13619 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8ef40897a7148b89e43cb5d562dee9a: Neither blocks nor log segments found. Creating new log.
I20260926 01:00:28.815755 13619 log.cc:824] T 00000000000000000000000000000000 P e8ef40897a7148b89e43cb5d562dee9a: Log is configured to *not* fsync() on all Append() calls
I20260926 01:00:28.816540 13619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8ef40897a7148b89e43cb5d562dee9a: No bootstrap required, opened a new log
I20260926 01:00:28.819075 13619 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8ef40897a7148b89e43cb5d562dee9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8ef40897a7148b89e43cb5d562dee9a" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 41719 } }
I20260926 01:00:28.819211 13619 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8ef40897a7148b89e43cb5d562dee9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:00:28.819233 13619 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8ef40897a7148b89e43cb5d562dee9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8ef40897a7148b89e43cb5d562dee9a, State: Initialized, Role: FOLLOWER
I20260926 01:00:28.819375 13619 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8ef40897a7148b89e43cb5d562dee9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8ef40897a7148b89e43cb5d562dee9a" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 41719 } }
Sep 26 01:00:28 dist-test-slave-7v10 krb5kdc[13538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 01:00:28.819455 13619 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8ef40897a7148b89e43cb5d562dee9a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:00:28.819495 13619 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8ef40897a7148b89e43cb5d562dee9a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:00:28.819554 13619 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8ef40897a7148b89e43cb5d562dee9a [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:00:28.820086 13619 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8ef40897a7148b89e43cb5d562dee9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8ef40897a7148b89e43cb5d562dee9a" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 41719 } }
I20260926 01:00:28.820173 13619 leader_election.cc:302] T 00000000000000000000000000000000 P e8ef40897a7148b89e43cb5d562dee9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8ef40897a7148b89e43cb5d562dee9a; no voters:
I20260926 01:00:28.820411 13619 leader_election.cc:288] T 00000000000000000000000000000000 P e8ef40897a7148b89e43cb5d562dee9a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:00:28.820446 13624 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8ef40897a7148b89e43cb5d562dee9a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:00:28.820614 13624 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8ef40897a7148b89e43cb5d562dee9a [term 1 LEADER]: Becoming Leader. State: Replica: e8ef40897a7148b89e43cb5d562dee9a, State: Running, Role: LEADER
I20260926 01:00:28.820712 13619 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8ef40897a7148b89e43cb5d562dee9a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:00:28.820719 13624 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8ef40897a7148b89e43cb5d562dee9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8ef40897a7148b89e43cb5d562dee9a" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 41719 } }
I20260926 01:00:28.821439 13625 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8ef40897a7148b89e43cb5d562dee9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8ef40897a7148b89e43cb5d562dee9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8ef40897a7148b89e43cb5d562dee9a" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 41719 } } }
I20260926 01:00:28.821460 13626 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8ef40897a7148b89e43cb5d562dee9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8ef40897a7148b89e43cb5d562dee9a. Latest consensus state: current_term: 1 leader_uuid: "e8ef40897a7148b89e43cb5d562dee9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8ef40897a7148b89e43cb5d562dee9a" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 41719 } } }
I20260926 01:00:28.821568 13625 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8ef40897a7148b89e43cb5d562dee9a [sys.catalog]: This master's current role is: LEADER
I20260926 01:00:28.821580 13626 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8ef40897a7148b89e43cb5d562dee9a [sys.catalog]: This master's current role is: LEADER
W20260926 01:00:28.822561 13639 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e8ef40897a7148b89e43cb5d562dee9a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:00:28.822788 13639 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:00:28.822904 13634 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:00:28.823092 13634 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:00:28.823863 13622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:28.813020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57254 (local address 127.3.75.62:41719)
0926 01:00:28.813197 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:28.813206 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:28.813236 (+ 30us) server_negotiation.cc:407] Connection header received
0926 01:00:28.813625 (+ 389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:28.813631 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:28.813676 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:28.813791 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:28.814588 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:28.815578 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:28.816623 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:28.817058 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:28.817140 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:28.819958 (+ 2818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:28.819962 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:28.819962 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:00:28.820003 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:28.822225 (+ 2222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:28.822617 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:28.822621 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:28.822622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:28.822733 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:28.822998 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:28.822999 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:28.822999 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:28.823291 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:28.823439 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:28.823559 (+ 120us) server_negotiation.cc:299] Negotiation successful
0926 01:00:28.823699 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":83,"threads_started":1}
I20260926 01:00:28.825075 13634 catalog_manager.cc:1347] Generated new cluster ID: d7e9c763e02d48e881cb21aedb641d06
I20260926 01:00:28.825129 13634 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:00:28.836694 13634 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:00:28.837209 13634 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:00:28.842034 13634 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8ef40897a7148b89e43cb5d562dee9a: Generated new TSK 0
I20260926 01:00:28.842221 13634 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.75.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab.
Entry for principal kudu/127.3.75.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab.
I20260926 01:00:28.869951 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.1:0
--local_ip_for_outbound_sockets=127.3.75.1
--webserver_interface=127.3.75.1
--webserver_port=0
--tserver_master_addrs=127.3.75.62:41719
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab
--principal=kudu/127.3.75.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:36085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:00:28.950724 13645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:00:28.950961 13645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:00:28.951000 13645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:00:28.952152 13645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:00:28.952214 13645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:00:28.952260 13645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.1
I20260926 01:00:28.953660 13645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:36085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab
--principal=kudu/127.3.75.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.75.1
--webserver_port=0
--tserver_master_addrs=127.3.75.62:41719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.1
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:00:28.953897 13645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:00:28.954157 13645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:00:28.959062 13651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:00:28.959131 13653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:00:28.959076 13650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:00:28.959342 13645 server_base.cc:1034] running on GCE node
I20260926 01:00:28.959527 13645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:00:28.959743 13645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:00:28.960894 13645 hybrid_clock.cc:648] HybridClock initialized: now 1790384428960875 us; error 35 us; skew 500 ppm
Sep 26 01:00:28 dist-test-slave-7v10 krb5kdc[13538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384428, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:00:28.963596 13645 init.cc:377] Logged in from keytab as kudu/127.3.75.1@KRBTEST.COM (short username kudu)
I20260926 01:00:28.964044 13645 webserver.cc:460] Webserver started at http://127.3.75.1:36371/ using document root <none> and password file <none>
I20260926 01:00:28.964269 13645 fs_manager.cc:359] Metadata directory not provided
I20260926 01:00:28.964318 13645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:00:28.964494 13645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:00:28.965549 13645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/ts-0/data/instance:
uuid: "01508e0867d94869be63a2b20f832d29"
format_stamp: "Formatted at 2026-09-26 01:00:28 on dist-test-slave-7v10"
server_key: "9f24a80cbc4037f402a27cfd60eb43f4"
server_key_iv: "8abc1355de2ca75d1f42b05ee92f4759"
server_key_version: "encryptionkey@0"
I20260926 01:00:28.965916 13645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/ts-0/wal/instance:
uuid: "01508e0867d94869be63a2b20f832d29"
format_stamp: "Formatted at 2026-09-26 01:00:28 on dist-test-slave-7v10"
server_key: "9f24a80cbc4037f402a27cfd60eb43f4"
server_key_iv: "8abc1355de2ca75d1f42b05ee92f4759"
server_key_version: "encryptionkey@0"
I20260926 01:00:28.967319 13645 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 01:00:28.968036 13660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:28.968212 13645 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 01:00:28.968307 13645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/ts-0/wal
uuid: "01508e0867d94869be63a2b20f832d29"
format_stamp: "Formatted at 2026-09-26 01:00:28 on dist-test-slave-7v10"
server_key: "9f24a80cbc4037f402a27cfd60eb43f4"
server_key_iv: "8abc1355de2ca75d1f42b05ee92f4759"
server_key_version: "encryptionkey@0"
I20260926 01:00:28.968389 13645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:00:28.982465 13645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:00:28.984404 13645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:00:28.984560 13645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:00:28.984786 13645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:00:28.985085 13645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:00:28.985118 13645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:28.985140 13645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:00:28.985185 13645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:28.990656 13645 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.1:33899
I20260926 01:00:28.990741 13774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.1:33899 every 8 connection(s)
I20260926 01:00:28.991056 13645 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
I20260926 01:00:28.994637 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 13645
I20260926 01:00:28.994719 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/ts-0/wal/instance
I20260926 01:00:28.994952 3372 external_mini_cluster.cc:1428] Setting key b50e8226966a1dde288856d74ac169de
Sep 26 01:00:28 dist-test-slave-7v10 krb5kdc[13538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384428, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.1@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 01:00:29.000675 13622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:28.991763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:33737 (local address 127.3.75.62:41719)
0926 01:00:28.991826 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:28.991828 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:28.992040 (+ 212us) server_negotiation.cc:407] Connection header received
0926 01:00:28.992592 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:28.992594 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:28.992606 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:28.992669 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:28.993618 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:28.994131 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:28.995399 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:28.995661 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:28.995734 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:28.997855 (+ 2121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:28.997856 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:28.997856 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:00:28.997863 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:28.999273 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:28.999698 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:28.999700 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:28.999700 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:28.999759 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:29.000003 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:29.000004 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:29.000004 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:29.000124 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:29.000200 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:29.000468 (+ 268us) server_negotiation.cc:299] Negotiation successful
0926 01:00:29.000573 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 01:00:29.000833 13777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:28.991734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:41719 (local address 127.3.75.1:33737)
0926 01:00:28.991903 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:28.991938 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:28.992482 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:28.992746 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:28.992749 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:28.993314 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:28.993502 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:28.993508 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:28.994230 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:28.994231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:28.995322 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:28.995326 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:28.995744 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:28.995745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:28.995846 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:28.996349 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:28.996357 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:28.997775 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:00:28.999375 (+ 1600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:28.999382 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:28.999384 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:28.999611 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:28.999825 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:28.999826 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:28.999827 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:28.999938 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:29.000190 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:00:29.000197 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:00:29.000338 (+ 141us) client_negotiation.cc:769] Sending connection context
0926 01:00:29.000435 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 01:00:29.000602 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":87,"threads_started":1}
I20260926 01:00:29.001313 13775 heartbeater.cc:347] Connected to a master server at 127.3.75.62:41719
I20260926 01:00:29.001405 13775 heartbeater.cc:464] Registering TS with master...
I20260926 01:00:29.001585 13775 heartbeater.cc:511] Master 127.3.75.62:41719 requested a full tablet report, sending...
I20260926 01:00:29.002144 13583 ts_manager.cc:194] Registered new tserver with Master: 01508e0867d94869be63a2b20f832d29 (127.3.75.1:33899)
I20260926 01:00:29.003172 13583 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.75.1@KRBTEST.COM'} at 127.3.75.1:33737
WARNING: no policy specified for kudu/127.3.75.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab.
Entry for principal kudu/127.3.75.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab.
I20260926 01:00:29.019799 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.2:0
--local_ip_for_outbound_sockets=127.3.75.2
--webserver_interface=127.3.75.2
--webserver_port=0
--tserver_master_addrs=127.3.75.62:41719
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab
--principal=kudu/127.3.75.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:36085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:00:29.099216 13780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:00:29.099416 13780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:00:29.099442 13780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:00:29.100577 13780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:00:29.100630 13780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:00:29.100656 13780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.2
I20260926 01:00:29.102029 13780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:36085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab
--principal=kudu/127.3.75.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.75.2
--webserver_port=0
--tserver_master_addrs=127.3.75.62:41719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.2
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:00:29.102211 13780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:00:29.102385 13780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:00:29.107311 13786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:00:29.107362 13780 server_base.cc:1034] running on GCE node
W20260926 01:00:29.107407 13788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:00:29.107404 13785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:00:29.107764 13780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:00:29.107995 13780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:00:29.109174 13780 hybrid_clock.cc:648] HybridClock initialized: now 1790384429109153 us; error 39 us; skew 500 ppm
Sep 26 01:00:29 dist-test-slave-7v10 krb5kdc[13538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384429, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:00:29.111789 13780 init.cc:377] Logged in from keytab as kudu/127.3.75.2@KRBTEST.COM (short username kudu)
I20260926 01:00:29.112243 13780 webserver.cc:460] Webserver started at http://127.3.75.2:44819/ using document root <none> and password file <none>
I20260926 01:00:29.112486 13780 fs_manager.cc:359] Metadata directory not provided
I20260926 01:00:29.112555 13780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:00:29.112697 13780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:00:29.113771 13780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/ts-1/data/instance:
uuid: "0656a89b1ef74d35a5f31b0498fced79"
format_stamp: "Formatted at 2026-09-26 01:00:29 on dist-test-slave-7v10"
server_key: "e25b798b1d4928b58eaeb5f887e2167c"
server_key_iv: "b86cce902b9bee9e8916028f3c36c909"
server_key_version: "encryptionkey@0"
I20260926 01:00:29.114142 13780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/ts-1/wal/instance:
uuid: "0656a89b1ef74d35a5f31b0498fced79"
format_stamp: "Formatted at 2026-09-26 01:00:29 on dist-test-slave-7v10"
server_key: "e25b798b1d4928b58eaeb5f887e2167c"
server_key_iv: "b86cce902b9bee9e8916028f3c36c909"
server_key_version: "encryptionkey@0"
I20260926 01:00:29.115550 13780 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 01:00:29.116271 13795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:29.116434 13780 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:00:29.116524 13780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/ts-1/wal
uuid: "0656a89b1ef74d35a5f31b0498fced79"
format_stamp: "Formatted at 2026-09-26 01:00:29 on dist-test-slave-7v10"
server_key: "e25b798b1d4928b58eaeb5f887e2167c"
server_key_iv: "b86cce902b9bee9e8916028f3c36c909"
server_key_version: "encryptionkey@0"
I20260926 01:00:29.116602 13780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:00:29.124684 13780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:00:29.126389 13780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:00:29.126539 13780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:00:29.126755 13780 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:00:29.127110 13780 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:00:29.127161 13780 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:29.127202 13780 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:00:29.127230 13780 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:29.132829 13780 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.2:33827
I20260926 01:00:29.132892 13910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.2:33827 every 8 connection(s)
I20260926 01:00:29.133235 13780 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
I20260926 01:00:29.134172 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 13780
I20260926 01:00:29.134280 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/ts-1/wal/instance
I20260926 01:00:29.134493 3372 external_mini_cluster.cc:1428] Setting key c87153a13763029fa4849fd2adc83c56
Sep 26 01:00:29 dist-test-slave-7v10 krb5kdc[13538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384429, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.2@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.75.3@KRBTEST.COM; defaulting to no policy
I20260926 01:00:29.144330 13622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:29.133933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:40913 (local address 127.3.75.62:41719)
0926 01:00:29.134027 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:29.134029 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:29.134254 (+ 225us) server_negotiation.cc:407] Connection header received
0926 01:00:29.134971 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:29.134972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:29.134982 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:29.135026 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:29.136319 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:29.137011 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:29.138008 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:29.138351 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:29.138429 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:29.140868 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:29.140870 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:29.140870 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:00:29.140879 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:29.142521 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:29.142967 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:29.142969 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:29.142969 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:29.143035 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:29.143361 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:29.143362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:29.143362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:29.143535 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:29.143608 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:29.144022 (+ 414us) server_negotiation.cc:299] Negotiation successful
0926 01:00:29.144190 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 01:00:29.144390 13913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:29.133899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:41719 (local address 127.3.75.2:40913)
0926 01:00:29.134095 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:29.134151 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:29.134806 (+ 655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:29.135096 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:29.135101 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:29.135935 (+ 834us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:29.136228 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:29.136235 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:29.137117 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:29.137118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:29.137914 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:29.137916 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:29.138437 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:29.138438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:29.138545 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:29.139129 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:29.139138 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:29.140771 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:00:29.142603 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:29.142609 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:29.142611 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:29.142836 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:29.143131 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:29.143132 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:29.143132 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:29.143293 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:29.143613 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:00:29.143620 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:00:29.143786 (+ 166us) client_negotiation.cc:769] Sending connection context
0926 01:00:29.143925 (+ 139us) client_negotiation.cc:240] Negotiation successful
0926 01:00:29.144144 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":106,"threads_started":1}
I20260926 01:00:29.145064 13911 heartbeater.cc:347] Connected to a master server at 127.3.75.62:41719
I20260926 01:00:29.145190 13911 heartbeater.cc:464] Registering TS with master...
I20260926 01:00:29.145401 13911 heartbeater.cc:511] Master 127.3.75.62:41719 requested a full tablet report, sending...
I20260926 01:00:29.145879 13583 ts_manager.cc:194] Registered new tserver with Master: 0656a89b1ef74d35a5f31b0498fced79 (127.3.75.2:33827)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.3@KRBTEST.COM" created.
I20260926 01:00:29.146401 13583 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.75.2@KRBTEST.COM'} at 127.3.75.2:40913
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab.
Entry for principal kudu/127.3.75.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab.
I20260926 01:00:29.159852 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.3:0
--local_ip_for_outbound_sockets=127.3.75.3
--webserver_interface=127.3.75.3
--webserver_port=0
--tserver_master_addrs=127.3.75.62:41719
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab
--principal=kudu/127.3.75.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:36085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:00:29.239872 13916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:00:29.240067 13916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:00:29.240088 13916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:00:29.241250 13916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:00:29.241291 13916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:00:29.241317 13916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.3
I20260926 01:00:29.242791 13916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:36085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab
--principal=kudu/127.3.75.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.75.3
--webserver_port=0
--tserver_master_addrs=127.3.75.62:41719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.3
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:00:29.243042 13916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:00:29.243288 13916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:00:29.248430 13924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:00:29.248587 13916 server_base.cc:1034] running on GCE node
W20260926 01:00:29.248474 13922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:00:29.248456 13921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:00:29.248886 13916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:00:29.249143 13916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:00:29.250298 13916 hybrid_clock.cc:648] HybridClock initialized: now 1790384429250285 us; error 41 us; skew 500 ppm
Sep 26 01:00:29 dist-test-slave-7v10 krb5kdc[13538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384429, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:00:29.252844 13916 init.cc:377] Logged in from keytab as kudu/127.3.75.3@KRBTEST.COM (short username kudu)
I20260926 01:00:29.253297 13916 webserver.cc:460] Webserver started at http://127.3.75.3:41097/ using document root <none> and password file <none>
I20260926 01:00:29.253551 13916 fs_manager.cc:359] Metadata directory not provided
I20260926 01:00:29.253616 13916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:00:29.253752 13916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:00:29.254767 13916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/ts-2/data/instance:
uuid: "c6bb326202c74c708b6ef454f29a749d"
format_stamp: "Formatted at 2026-09-26 01:00:29 on dist-test-slave-7v10"
server_key: "1a80aed7030c717b817f6735e0b0c7e2"
server_key_iv: "a333f898b785fd79b9638b5f9d3b6134"
server_key_version: "encryptionkey@0"
I20260926 01:00:29.255162 13916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/ts-2/wal/instance:
uuid: "c6bb326202c74c708b6ef454f29a749d"
format_stamp: "Formatted at 2026-09-26 01:00:29 on dist-test-slave-7v10"
server_key: "1a80aed7030c717b817f6735e0b0c7e2"
server_key_iv: "a333f898b785fd79b9638b5f9d3b6134"
server_key_version: "encryptionkey@0"
I20260926 01:00:29.256433 13916 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:00:29.257150 13931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:29.257314 13916 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:00:29.257406 13916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/ts-2/wal
uuid: "c6bb326202c74c708b6ef454f29a749d"
format_stamp: "Formatted at 2026-09-26 01:00:29 on dist-test-slave-7v10"
server_key: "1a80aed7030c717b817f6735e0b0c7e2"
server_key_iv: "a333f898b785fd79b9638b5f9d3b6134"
server_key_version: "encryptionkey@0"
I20260926 01:00:29.257479 13916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:00:29.276046 13916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:00:29.277954 13916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:00:29.278122 13916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:00:29.278354 13916 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:00:29.278678 13916 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:00:29.278730 13916 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:29.278772 13916 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:00:29.278798 13916 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:00:29.284708 13916 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.3:32813
I20260926 01:00:29.284785 14044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.3:32813 every 8 connection(s)
I20260926 01:00:29.285117 13916 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
Sep 26 01:00:29 dist-test-slave-7v10 krb5kdc[13538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384429, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.3@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 01:00:29.294104 13622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:29.285893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.3:51407 (local address 127.3.75.62:41719)
0926 01:00:29.285946 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:29.285948 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:29.286131 (+ 183us) server_negotiation.cc:407] Connection header received
0926 01:00:29.286664 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:29.286666 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:29.286679 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:29.286732 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:29.287710 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:29.288205 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:29.289198 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:29.289392 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:29.289474 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:29.291654 (+ 2180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:29.291655 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:29.291656 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:00:29.291663 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:29.292799 (+ 1136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:29.293192 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:29.293193 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:29.293194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:29.293249 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:29.293487 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:29.293488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:29.293488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:29.293599 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:29.293670 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:29.293928 (+ 258us) server_negotiation.cc:299] Negotiation successful
0926 01:00:29.294015 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 01:00:29.294293 14047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:29.285854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:41719 (local address 127.3.75.3:51407)
0926 01:00:29.285990 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:29.286038 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:29.286556 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:29.286799 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:29.286803 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:29.287389 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:00:29.287586 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:29.287591 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:29.288297 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:29.288299 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:29.289107 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:29.289109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:29.289489 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:29.289490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:29.289611 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:29.290181 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:00:29.290189 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:00:29.291562 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:00:29.292886 (+ 1324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:29.292891 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:29.292894 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:29.293107 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:29.293313 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:00:29.293314 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:00:29.293314 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:00:29.293412 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:00:29.293661 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:00:29.293667 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:00:29.293800 (+ 133us) client_negotiation.cc:769] Sending connection context
0926 01:00:29.293906 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 01:00:29.294057 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":60,"threads_started":1}
I20260926 01:00:29.294804 14045 heartbeater.cc:347] Connected to a master server at 127.3.75.62:41719
I20260926 01:00:29.294824 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 13916
I20260926 01:00:29.294921 14045 heartbeater.cc:464] Registering TS with master...
I20260926 01:00:29.294947 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/ts-2/wal/instance
I20260926 01:00:29.295107 14045 heartbeater.cc:511] Master 127.3.75.62:41719 requested a full tablet report, sending...
I20260926 01:00:29.295140 3372 external_mini_cluster.cc:1428] Setting key 30aa84fd29265b51ab554d1fca9aedc8
I20260926 01:00:29.295547 13582 ts_manager.cc:194] Registered new tserver with Master: c6bb326202c74c708b6ef454f29a749d (127.3.75.3:32813)
I20260926 01:00:29.295913 13582 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.75.3@KRBTEST.COM'} at 127.3.75.3:51407
I20260926 01:00:29.296592 3372 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 01:00:29.303529 13622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:29.297755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57270 (local address 127.3.75.62:41719)
0926 01:00:29.297790 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:29.297792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:29.297912 (+ 120us) server_negotiation.cc:407] Connection header received
0926 01:00:29.297986 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:29.297987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:29.297996 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:29.298069 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:00:29.298629 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:29.299169 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:29.299931 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:29.300125 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:29.300197 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:29.301270 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:00:29.301271 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:00:29.301272 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:00:29.301279 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:00:29.302372 (+ 1093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:29.302729 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:29.302730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:29.302730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:29.302784 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:00:29.302998 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:00:29.302998 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:00:29.302999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:00:29.303136 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:00:29.303238 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:29.303349 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 01:00:29.303420 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 01:00:29.304931 13582 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57270:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:00:29.306401 13582 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:00:29.312486 14055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:29.308321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.2:33827 (local address 127.0.0.1:45500)
0926 01:00:29.308513 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:29.308553 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:29.308618 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:29.309093 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:29.309094 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:29.309096 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:00:29.309130 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:29.309131 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:29.310415 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:29.310416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:29.311239 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:29.311241 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:29.312172 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:29.312174 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:29.312266 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:29.312280 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 01:00:29.312350 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 01:00:29.312411 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":48,"threads_started":1}
I20260926 01:00:29.312536 14056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:29.308407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45500 (local address 127.3.75.2:33827)
0926 01:00:29.308885 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:29.308907 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:29.308916 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:00:29.308951 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:29.308953 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:29.308968 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:29.309226 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:00:29.309303 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:29.310324 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:29.311317 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:29.312082 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:29.312164 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:29.312226 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:29.312372 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 01:00:29.312420 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":93,"threads_started":1}
I20260926 01:00:29.312577 14059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:29.308718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41806 (local address 127.3.75.1:33899)
0926 01:00:29.308892 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:29.308897 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:29.308906 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:00:29.308945 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:29.308947 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:29.308959 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:29.309036 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:00:29.309157 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:29.309945 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:29.311618 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:29.312120 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:29.312369 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:29.312427 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:29.312451 (+ 24us) server_negotiation.cc:299] Negotiation successful
0926 01:00:29.312485 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":75,"threads_started":1}
I20260926 01:00:29.312745 14058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:29.308504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.1:33899 (local address 127.0.0.1:41806)
0926 01:00:29.308620 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:29.308629 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:29.308669 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:29.309046 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:29.309047 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:29.309050 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:00:29.309103 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:29.309105 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:29.310052 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:29.310053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:29.311547 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:29.311548 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:29.312244 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:29.312245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:29.312290 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:29.312292 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 01:00:29.312326 (+ 34us) client_negotiation.cc:240] Negotiation successful
0926 01:00:29.312691 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":315,"thread_start_us":74,"threads_started":1}
I20260926 01:00:29.312809 14057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:29.308439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38466 (local address 127.3.75.3:32813)
0926 01:00:29.308748 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:29.308753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:29.308762 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:00:29.308818 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:29.308821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:29.308836 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:29.308921 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:00:29.309101 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:29.310207 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:29.311451 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:29.312131 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:29.312189 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:29.312245 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:29.312523 (+ 278us) server_negotiation.cc:299] Negotiation successful
0926 01:00:29.312723 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":71,"threads_started":1}
I20260926 01:00:29.312956 14054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:29.308369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.3:32813 (local address 127.0.0.1:38466)
0926 01:00:29.308583 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:29.308594 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:29.308647 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:29.308948 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:29.308952 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:29.308954 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:00:29.309017 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:29.309019 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:29.310304 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:29.310305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:29.311377 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:29.311379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:29.312283 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:29.312284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:29.312375 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:29.312380 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 01:00:29.312428 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 01:00:29.312465 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":62,"threads_started":1}
I20260926 01:00:29.313436 13844 tablet_service.cc:1467] Processing CreateTablet for tablet d7526f3459eb4a029b0144bb853cac79 (DEFAULT_TABLE table=test-table [id=66fc6263595747ae916269ed592d0e8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:00:29.313454 13708 tablet_service.cc:1467] Processing CreateTablet for tablet d7526f3459eb4a029b0144bb853cac79 (DEFAULT_TABLE table=test-table [id=66fc6263595747ae916269ed592d0e8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:00:29.313804 13708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7526f3459eb4a029b0144bb853cac79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:00:29.313804 13844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7526f3459eb4a029b0144bb853cac79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:00:29.315447 13979 tablet_service.cc:1467] Processing CreateTablet for tablet d7526f3459eb4a029b0144bb853cac79 (DEFAULT_TABLE table=test-table [id=66fc6263595747ae916269ed592d0e8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:00:29.315730 13979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7526f3459eb4a029b0144bb853cac79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:00:29.317188 14060 tablet_bootstrap.cc:492] T d7526f3459eb4a029b0144bb853cac79 P 01508e0867d94869be63a2b20f832d29: Bootstrap starting.
I20260926 01:00:29.317902 14060 tablet_bootstrap.cc:654] T d7526f3459eb4a029b0144bb853cac79 P 01508e0867d94869be63a2b20f832d29: Neither blocks nor log segments found. Creating new log.
I20260926 01:00:29.318492 14060 log.cc:824] T d7526f3459eb4a029b0144bb853cac79 P 01508e0867d94869be63a2b20f832d29: Log is configured to *not* fsync() on all Append() calls
I20260926 01:00:29.318868 14061 tablet_bootstrap.cc:492] T d7526f3459eb4a029b0144bb853cac79 P 0656a89b1ef74d35a5f31b0498fced79: Bootstrap starting.
I20260926 01:00:29.319139 14063 tablet_bootstrap.cc:492] T d7526f3459eb4a029b0144bb853cac79 P c6bb326202c74c708b6ef454f29a749d: Bootstrap starting.
I20260926 01:00:29.319366 14060 tablet_bootstrap.cc:492] T d7526f3459eb4a029b0144bb853cac79 P 01508e0867d94869be63a2b20f832d29: No bootstrap required, opened a new log
I20260926 01:00:29.319435 14060 ts_tablet_manager.cc:1397] T d7526f3459eb4a029b0144bb853cac79 P 01508e0867d94869be63a2b20f832d29: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 01:00:29.319609 14061 tablet_bootstrap.cc:654] T d7526f3459eb4a029b0144bb853cac79 P 0656a89b1ef74d35a5f31b0498fced79: Neither blocks nor log segments found. Creating new log.
I20260926 01:00:29.319949 14063 tablet_bootstrap.cc:654] T d7526f3459eb4a029b0144bb853cac79 P c6bb326202c74c708b6ef454f29a749d: Neither blocks nor log segments found. Creating new log.
I20260926 01:00:29.320127 14061 log.cc:824] T d7526f3459eb4a029b0144bb853cac79 P 0656a89b1ef74d35a5f31b0498fced79: Log is configured to *not* fsync() on all Append() calls
I20260926 01:00:29.320561 14063 log.cc:824] T d7526f3459eb4a029b0144bb853cac79 P c6bb326202c74c708b6ef454f29a749d: Log is configured to *not* fsync() on all Append() calls
I20260926 01:00:29.320956 14061 tablet_bootstrap.cc:492] T d7526f3459eb4a029b0144bb853cac79 P 0656a89b1ef74d35a5f31b0498fced79: No bootstrap required, opened a new log
I20260926 01:00:29.321031 14061 ts_tablet_manager.cc:1397] T d7526f3459eb4a029b0144bb853cac79 P 0656a89b1ef74d35a5f31b0498fced79: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 01:00:29.321345 14063 tablet_bootstrap.cc:492] T d7526f3459eb4a029b0144bb853cac79 P c6bb326202c74c708b6ef454f29a749d: No bootstrap required, opened a new log
I20260926 01:00:29.321560 14063 ts_tablet_manager.cc:1397] T d7526f3459eb4a029b0144bb853cac79 P c6bb326202c74c708b6ef454f29a749d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 01:00:29.321754 14060 raft_consensus.cc:348] T d7526f3459eb4a029b0144bb853cac79 P 01508e0867d94869be63a2b20f832d29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0656a89b1ef74d35a5f31b0498fced79" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 33827 } } peers { permanent_uuid: "01508e0867d94869be63a2b20f832d29" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 33899 } } peers { permanent_uuid: "c6bb326202c74c708b6ef454f29a749d" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 32813 } }
I20260926 01:00:29.321926 14060 raft_consensus.cc:374] T d7526f3459eb4a029b0144bb853cac79 P 01508e0867d94869be63a2b20f832d29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:00:29.322026 14060 raft_consensus.cc:729] T d7526f3459eb4a029b0144bb853cac79 P 01508e0867d94869be63a2b20f832d29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01508e0867d94869be63a2b20f832d29, State: Initialized, Role: FOLLOWER
I20260926 01:00:29.322147 14060 consensus_queue.cc:260] T d7526f3459eb4a029b0144bb853cac79 P 01508e0867d94869be63a2b20f832d29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0656a89b1ef74d35a5f31b0498fced79" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 33827 } } peers { permanent_uuid: "01508e0867d94869be63a2b20f832d29" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 33899 } } peers { permanent_uuid: "c6bb326202c74c708b6ef454f29a749d" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 32813 } }
I20260926 01:00:29.322415 14060 ts_tablet_manager.cc:1428] T d7526f3459eb4a029b0144bb853cac79 P 01508e0867d94869be63a2b20f832d29: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 01:00:29.322590 13775 heartbeater.cc:503] Master 127.3.75.62:41719 was elected leader, sending a full tablet report...
I20260926 01:00:29.323097 14061 raft_consensus.cc:348] T d7526f3459eb4a029b0144bb853cac79 P 0656a89b1ef74d35a5f31b0498fced79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0656a89b1ef74d35a5f31b0498fced79" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 33827 } } peers { permanent_uuid: "01508e0867d94869be63a2b20f832d29" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 33899 } } peers { permanent_uuid: "c6bb326202c74c708b6ef454f29a749d" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 32813 } }
I20260926 01:00:29.323244 14061 raft_consensus.cc:374] T d7526f3459eb4a029b0144bb853cac79 P 0656a89b1ef74d35a5f31b0498fced79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:00:29.323284 14061 raft_consensus.cc:729] T d7526f3459eb4a029b0144bb853cac79 P 0656a89b1ef74d35a5f31b0498fced79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0656a89b1ef74d35a5f31b0498fced79, State: Initialized, Role: FOLLOWER
I20260926 01:00:29.323421 14061 consensus_queue.cc:260] T d7526f3459eb4a029b0144bb853cac79 P 0656a89b1ef74d35a5f31b0498fced79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0656a89b1ef74d35a5f31b0498fced79" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 33827 } } peers { permanent_uuid: "01508e0867d94869be63a2b20f832d29" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 33899 } } peers { permanent_uuid: "c6bb326202c74c708b6ef454f29a749d" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 32813 } }
I20260926 01:00:29.323676 14061 ts_tablet_manager.cc:1428] T d7526f3459eb4a029b0144bb853cac79 P 0656a89b1ef74d35a5f31b0498fced79: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260926 01:00:29.323701 13911 heartbeater.cc:503] Master 127.3.75.62:41719 was elected leader, sending a full tablet report...
I20260926 01:00:29.323930 14063 raft_consensus.cc:348] T d7526f3459eb4a029b0144bb853cac79 P c6bb326202c74c708b6ef454f29a749d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0656a89b1ef74d35a5f31b0498fced79" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 33827 } } peers { permanent_uuid: "01508e0867d94869be63a2b20f832d29" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 33899 } } peers { permanent_uuid: "c6bb326202c74c708b6ef454f29a749d" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 32813 } }
I20260926 01:00:29.324100 14063 raft_consensus.cc:374] T d7526f3459eb4a029b0144bb853cac79 P c6bb326202c74c708b6ef454f29a749d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:00:29.324144 14063 raft_consensus.cc:729] T d7526f3459eb4a029b0144bb853cac79 P c6bb326202c74c708b6ef454f29a749d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6bb326202c74c708b6ef454f29a749d, State: Initialized, Role: FOLLOWER
I20260926 01:00:29.324249 14063 consensus_queue.cc:260] T d7526f3459eb4a029b0144bb853cac79 P c6bb326202c74c708b6ef454f29a749d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0656a89b1ef74d35a5f31b0498fced79" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 33827 } } peers { permanent_uuid: "01508e0867d94869be63a2b20f832d29" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 33899 } } peers { permanent_uuid: "c6bb326202c74c708b6ef454f29a749d" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 32813 } }
I20260926 01:00:29.324528 14063 ts_tablet_manager.cc:1428] T d7526f3459eb4a029b0144bb853cac79 P c6bb326202c74c708b6ef454f29a749d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 01:00:29.324649 14045 heartbeater.cc:503] Master 127.3.75.62:41719 was elected leader, sending a full tablet report...
W20260926 01:00:29.383852 13912 tablet.cc:2369] T d7526f3459eb4a029b0144bb853cac79 P 0656a89b1ef74d35a5f31b0498fced79: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 01:00:29.491739 13776 tablet.cc:2369] T d7526f3459eb4a029b0144bb853cac79 P 01508e0867d94869be63a2b20f832d29: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 01:00:29.535717 14046 tablet.cc:2369] T d7526f3459eb4a029b0144bb853cac79 P c6bb326202c74c708b6ef454f29a749d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:00:29.668749 14067 raft_consensus.cc:482] T d7526f3459eb4a029b0144bb853cac79 P 0656a89b1ef74d35a5f31b0498fced79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:00:29.668876 14067 raft_consensus.cc:504] T d7526f3459eb4a029b0144bb853cac79 P 0656a89b1ef74d35a5f31b0498fced79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0656a89b1ef74d35a5f31b0498fced79" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 33827 } } peers { permanent_uuid: "01508e0867d94869be63a2b20f832d29" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 33899 } } peers { permanent_uuid: "c6bb326202c74c708b6ef454f29a749d" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 32813 } }
I20260926 01:00:29.669312 14067 leader_election.cc:288] T d7526f3459eb4a029b0144bb853cac79 P 0656a89b1ef74d35a5f31b0498fced79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 01508e0867d94869be63a2b20f832d29 (127.3.75.1:33899), c6bb326202c74c708b6ef454f29a749d (127.3.75.3:32813)
I20260926 01:00:29.672353 14057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:29.669607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:51653 (local address 127.3.75.3:32813)
0926 01:00:29.669655 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:29.669657 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:29.669708 (+ 51us) server_negotiation.cc:407] Connection header received
0926 01:00:29.669788 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:29.669789 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:29.669802 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:29.669874 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:00:29.670035 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:29.670575 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:29.671626 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:29.672009 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:29.672094 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:29.672104 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:29.672250 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 01:00:29.672288 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 01:00:29.672372 14069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:29.669499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.3:32813 (local address 127.3.75.2:51653)
0926 01:00:29.669650 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:29.669663 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:29.669725 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:29.669878 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:29.669879 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:29.669881 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:00:29.669952 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:29.669954 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:29.670651 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:29.670652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:29.671549 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:29.671552 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:29.672094 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:29.672095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:29.672168 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:29.672172 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:00:29.672242 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 01:00:29.672284 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":79,"threads_started":1}
I20260926 01:00:29.672431 14059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:29.669623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:36167 (local address 127.3.75.1:33899)
0926 01:00:29.669670 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:29.669672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:29.669740 (+ 68us) server_negotiation.cc:407] Connection header received
0926 01:00:29.669832 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:29.669834 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:29.669844 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:29.669912 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:00:29.670074 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:29.670618 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:29.671653 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:29.672084 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:29.672170 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:29.672179 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:29.672328 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 01:00:29.672362 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 01:00:29.672479 14070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:29.669516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.1:33899 (local address 127.3.75.2:36167)
0926 01:00:29.669668 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:29.669678 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:00:29.669756 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:00:29.669915 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:00:29.669915 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:00:29.669917 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:00:29.670001 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:29.670003 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:29.670691 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:29.670692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:29.671566 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:00:29.671568 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:00:29.672174 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:00:29.672175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:00:29.672244 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:29.672248 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:00:29.672321 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 01:00:29.672359 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":20,"thread_start_us":82,"threads_started":1}
I20260926 01:00:29.672715 13999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7526f3459eb4a029b0144bb853cac79" candidate_uuid: "0656a89b1ef74d35a5f31b0498fced79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6bb326202c74c708b6ef454f29a749d" is_pre_election: true
I20260926 01:00:29.672863 13999 raft_consensus.cc:2454] T d7526f3459eb4a029b0144bb853cac79 P c6bb326202c74c708b6ef454f29a749d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0656a89b1ef74d35a5f31b0498fced79 in term 0.
I20260926 01:00:29.673126 13797 leader_election.cc:302] T d7526f3459eb4a029b0144bb853cac79 P 0656a89b1ef74d35a5f31b0498fced79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0656a89b1ef74d35a5f31b0498fced79, c6bb326202c74c708b6ef454f29a749d; no voters:
I20260926 01:00:29.673159 13728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7526f3459eb4a029b0144bb853cac79" candidate_uuid: "0656a89b1ef74d35a5f31b0498fced79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01508e0867d94869be63a2b20f832d29" is_pre_election: true
I20260926 01:00:29.673302 13728 raft_consensus.cc:2454] T d7526f3459eb4a029b0144bb853cac79 P 01508e0867d94869be63a2b20f832d29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0656a89b1ef74d35a5f31b0498fced79 in term 0.
I20260926 01:00:29.673331 14067 raft_consensus.cc:2781] T d7526f3459eb4a029b0144bb853cac79 P 0656a89b1ef74d35a5f31b0498fced79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:00:29.673395 14067 raft_consensus.cc:482] T d7526f3459eb4a029b0144bb853cac79 P 0656a89b1ef74d35a5f31b0498fced79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:00:29.673421 14067 raft_consensus.cc:3037] T d7526f3459eb4a029b0144bb853cac79 P 0656a89b1ef74d35a5f31b0498fced79 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:00:29.674190 14067 raft_consensus.cc:504] T d7526f3459eb4a029b0144bb853cac79 P 0656a89b1ef74d35a5f31b0498fced79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0656a89b1ef74d35a5f31b0498fced79" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 33827 } } peers { permanent_uuid: "01508e0867d94869be63a2b20f832d29" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 33899 } } peers { permanent_uuid: "c6bb326202c74c708b6ef454f29a749d" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 32813 } }
I20260926 01:00:29.674329 14067 leader_election.cc:288] T d7526f3459eb4a029b0144bb853cac79 P 0656a89b1ef74d35a5f31b0498fced79 [CANDIDATE]: Term 1 election: Requested vote from peers 01508e0867d94869be63a2b20f832d29 (127.3.75.1:33899), c6bb326202c74c708b6ef454f29a749d (127.3.75.3:32813)
I20260926 01:00:29.674485 13999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7526f3459eb4a029b0144bb853cac79" candidate_uuid: "0656a89b1ef74d35a5f31b0498fced79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6bb326202c74c708b6ef454f29a749d"
I20260926 01:00:29.674512 13728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7526f3459eb4a029b0144bb853cac79" candidate_uuid: "0656a89b1ef74d35a5f31b0498fced79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01508e0867d94869be63a2b20f832d29"
I20260926 01:00:29.674567 13999 raft_consensus.cc:3037] T d7526f3459eb4a029b0144bb853cac79 P c6bb326202c74c708b6ef454f29a749d [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:00:29.674574 13728 raft_consensus.cc:3037] T d7526f3459eb4a029b0144bb853cac79 P 01508e0867d94869be63a2b20f832d29 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:00:29.675453 13999 raft_consensus.cc:2454] T d7526f3459eb4a029b0144bb853cac79 P c6bb326202c74c708b6ef454f29a749d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0656a89b1ef74d35a5f31b0498fced79 in term 1.
I20260926 01:00:29.675453 13728 raft_consensus.cc:2454] T d7526f3459eb4a029b0144bb853cac79 P 01508e0867d94869be63a2b20f832d29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0656a89b1ef74d35a5f31b0498fced79 in term 1.
I20260926 01:00:29.675628 13797 leader_election.cc:302] T d7526f3459eb4a029b0144bb853cac79 P 0656a89b1ef74d35a5f31b0498fced79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0656a89b1ef74d35a5f31b0498fced79, c6bb326202c74c708b6ef454f29a749d; no voters:
I20260926 01:00:29.675725 14067 raft_consensus.cc:2781] T d7526f3459eb4a029b0144bb853cac79 P 0656a89b1ef74d35a5f31b0498fced79 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:00:29.675873 14067 raft_consensus.cc:686] T d7526f3459eb4a029b0144bb853cac79 P 0656a89b1ef74d35a5f31b0498fced79 [term 1 LEADER]: Becoming Leader. State: Replica: 0656a89b1ef74d35a5f31b0498fced79, State: Running, Role: LEADER
I20260926 01:00:29.675998 14067 consensus_queue.cc:237] T d7526f3459eb4a029b0144bb853cac79 P 0656a89b1ef74d35a5f31b0498fced79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0656a89b1ef74d35a5f31b0498fced79" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 33827 } } peers { permanent_uuid: "01508e0867d94869be63a2b20f832d29" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 33899 } } peers { permanent_uuid: "c6bb326202c74c708b6ef454f29a749d" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 32813 } }
I20260926 01:00:29.676936 13582 catalog_manager.cc:5495] T d7526f3459eb4a029b0144bb853cac79 P 0656a89b1ef74d35a5f31b0498fced79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0656a89b1ef74d35a5f31b0498fced79 (127.3.75.2). New cstate: current_term: 1 leader_uuid: "0656a89b1ef74d35a5f31b0498fced79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0656a89b1ef74d35a5f31b0498fced79" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 33827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01508e0867d94869be63a2b20f832d29" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 33899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6bb326202c74c708b6ef454f29a749d" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 32813 } health_report { overall_health: UNKNOWN } } }
I20260926 01:00:29.754328 14056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:29.751886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45502 (local address 127.3.75.2:33827)
0926 01:00:29.751941 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 01:00:29.751944 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:00:29.751953 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:00:29.752033 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:00:29.752033 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:00:29.752054 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:00:29.752135 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:00:29.752297 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:29.752766 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:29.753548 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:00:29.753739 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:00:29.753806 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:00:29.753985 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:00:29.754099 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:00:29.754194 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 01:00:29.754214 (+ 20us) server_negotiation.cc:299] Negotiation successful
0926 01:00:29.754235 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 01:00:29.756193 13728 raft_consensus.cc:1261] T d7526f3459eb4a029b0144bb853cac79 P 01508e0867d94869be63a2b20f832d29 [term 1 FOLLOWER]: Refusing update from remote peer 0656a89b1ef74d35a5f31b0498fced79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:00:29.756198 13999 raft_consensus.cc:1261] T d7526f3459eb4a029b0144bb853cac79 P c6bb326202c74c708b6ef454f29a749d [term 1 FOLLOWER]: Refusing update from remote peer 0656a89b1ef74d35a5f31b0498fced79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:00:29.756584 14067 consensus_queue.cc:1035] T d7526f3459eb4a029b0144bb853cac79 P 0656a89b1ef74d35a5f31b0498fced79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6bb326202c74c708b6ef454f29a749d" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 32813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:00:29.756742 14071 consensus_queue.cc:1035] T d7526f3459eb4a029b0144bb853cac79 P 0656a89b1ef74d35a5f31b0498fced79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01508e0867d94869be63a2b20f832d29" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:00:29.759215 14076 mvcc.cc:205] Tried to move back new op lower bound from 7333414624278466560 to 7333414623953502208. Current Snapshot: MvccSnapshot[applied={T|T < 7333414624278466560}]
I20260926 01:00:29.761852 13583 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57270:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:00:29.761945 13583 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57270:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:00:29.762686 13583 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e8ef40897a7148b89e43cb5d562dee9a: Sending DeleteTablet for 3 replicas of tablet d7526f3459eb4a029b0144bb853cac79
I20260926 01:00:29.763069 13979 tablet_service.cc:1514] Processing DeleteTablet for tablet d7526f3459eb4a029b0144bb853cac79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:00:29 UTC) from {username='kudu', principal='kudu/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:38466
I20260926 01:00:29.763083 13708 tablet_service.cc:1514] Processing DeleteTablet for tablet d7526f3459eb4a029b0144bb853cac79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:00:29 UTC) from {username='kudu', principal='kudu/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:41806
I20260926 01:00:29.763325 14084 tablet_replica.cc:330] T d7526f3459eb4a029b0144bb853cac79 P c6bb326202c74c708b6ef454f29a749d: stopping tablet replica
I20260926 01:00:29.763334 14083 tablet_replica.cc:330] T d7526f3459eb4a029b0144bb853cac79 P 01508e0867d94869be63a2b20f832d29: stopping tablet replica
I20260926 01:00:29.763434 14083 raft_consensus.cc:2229] T d7526f3459eb4a029b0144bb853cac79 P 01508e0867d94869be63a2b20f832d29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:00:29.763458 14084 raft_consensus.cc:2229] T d7526f3459eb4a029b0144bb853cac79 P c6bb326202c74c708b6ef454f29a749d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:00:29.763563 14083 raft_consensus.cc:2258] T d7526f3459eb4a029b0144bb853cac79 P 01508e0867d94869be63a2b20f832d29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:00:29.763597 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 13645
I20260926 01:00:29.763728 14084 raft_consensus.cc:2258] T d7526f3459eb4a029b0144bb853cac79 P c6bb326202c74c708b6ef454f29a749d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:00:29.764405 14084 ts_tablet_manager.cc:1905] T d7526f3459eb4a029b0144bb853cac79 P c6bb326202c74c708b6ef454f29a749d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:00:29.765851 14084 ts_tablet_manager.cc:1918] T d7526f3459eb4a029b0144bb853cac79 P c6bb326202c74c708b6ef454f29a749d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:00:29.765933 14084 log.cc:1196] T d7526f3459eb4a029b0144bb853cac79 P c6bb326202c74c708b6ef454f29a749d: Deleting WAL directory at /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/ts-2/wal/wals/d7526f3459eb4a029b0144bb853cac79
I20260926 01:00:29.766170 14084 ts_tablet_manager.cc:1939] T d7526f3459eb4a029b0144bb853cac79 P c6bb326202c74c708b6ef454f29a749d: Deleting consensus metadata
I20260926 01:00:29.766554 13568 catalog_manager.cc:4841] TS c6bb326202c74c708b6ef454f29a749d (127.3.75.3:32813): tablet d7526f3459eb4a029b0144bb853cac79 (table test-table [id=66fc6263595747ae916269ed592d0e8a]) successfully deleted
I20260926 01:00:29.767525 13844 tablet_service.cc:1514] Processing DeleteTablet for tablet d7526f3459eb4a029b0144bb853cac79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:00:29 UTC) from {username='kudu', principal='kudu/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:45500
I20260926 01:00:29.767716 14085 tablet_replica.cc:330] T d7526f3459eb4a029b0144bb853cac79 P 0656a89b1ef74d35a5f31b0498fced79: stopping tablet replica
I20260926 01:00:29.767809 14085 raft_consensus.cc:2229] T d7526f3459eb4a029b0144bb853cac79 P 0656a89b1ef74d35a5f31b0498fced79 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:00:29.767941 14085 raft_consensus.cc:2258] T d7526f3459eb4a029b0144bb853cac79 P 0656a89b1ef74d35a5f31b0498fced79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:00:29.768321 14085 ts_tablet_manager.cc:1905] T d7526f3459eb4a029b0144bb853cac79 P 0656a89b1ef74d35a5f31b0498fced79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:00:29.769546 14085 ts_tablet_manager.cc:1918] T d7526f3459eb4a029b0144bb853cac79 P 0656a89b1ef74d35a5f31b0498fced79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:00:29.769627 14085 log.cc:1196] T d7526f3459eb4a029b0144bb853cac79 P 0656a89b1ef74d35a5f31b0498fced79: Deleting WAL directory at /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384372073861-3372-0/minicluster-data/ts-1/wal/wals/d7526f3459eb4a029b0144bb853cac79
I20260926 01:00:29.769848 14085 ts_tablet_manager.cc:1939] T d7526f3459eb4a029b0144bb853cac79 P 0656a89b1ef74d35a5f31b0498fced79: Deleting consensus metadata
I20260926 01:00:29.770206 13569 catalog_manager.cc:4841] TS 0656a89b1ef74d35a5f31b0498fced79 (127.3.75.2:33827): tablet d7526f3459eb4a029b0144bb853cac79 (table test-table [id=66fc6263595747ae916269ed592d0e8a]) successfully deleted
W20260926 01:00:29.771956 13569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.75.1:33899 (error 108)
I20260926 01:00:29.772137 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 13780
I20260926 01:00:29.772794 14054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:00:29.772255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.1:33899 (local address 127.0.0.1:41816)
0926 01:00:29.772730 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0926 01:00:29.772769 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.1:33899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457}
W20260926 01:00:29.772938 13569 catalog_manager.cc:4584] TS 01508e0867d94869be63a2b20f832d29 (127.3.75.1:33899): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d7526f3459eb4a029b0144bb853cac79: Network error: Client connection negotiation failed: client connection to 127.3.75.1:33899: connect: Connection refused (error 111)
I20260926 01:00:29.778362 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 13916
I20260926 01:00:29.782788 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 13551
2026-09-26T01:00:29Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2254 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:00:29 dist-test-slave-7v10 krb5kdc[14089](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:00:29 dist-test-slave-7v10 krb5kdc[14089](info): set up 2 sockets
Sep 26 01:00:29 dist-test-slave-7v10 krb5kdc[14089](info): commencing operation
krb5kdc: starting...
W20260926 01:00:30.827113 3372 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Sep 26 01:00:30 dist-test-slave-7v10 krb5kdc[14089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:00:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:00:30Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.75.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.75.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.75.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/krb5kdc/rangeradmin_127.3.75.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.75.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/krb5kdc/rangeradmin_127.3.75.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.75.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.75.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.75.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/krb5kdc/rangerlookup_127.3.75.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.75.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/krb5kdc/rangerlookup_127.3.75.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.75.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.75.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.75.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/krb5kdc/HTTP_127.3.75.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.75.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/krb5kdc/HTTP_127.3.75.20@KRBTEST.COM.keytab.
I20260926 01:00:31.007194 3372 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasktuUsJs/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/postgres -l logfile start
2026-09-26 01:00:33.442 UTC [14119] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 01:00:33.442 UTC [14119] LOG: listening on IPv4 address "127.3.75.20", port 43621
2026-09-26 01:00:33.445 UTC [14119] LOG: listening on Unix socket "/tmp/.s.PGSQL.43621"
2026-09-26 01:00:33.480 UTC [14122] LOG: database system was shut down at 2026-09-26 01:00:32 UTC
2026-09-26 01:00:33.485 UTC [14119] LOG: database system is ready to accept connections
I20260926 01:00:34.443013 3372 mini_postgres.cc:96] Postgres bound to 43621
2026-09-26 01:00:34.448 UTC [14130] FATAL: database "slave" does not exist
127.3.75.20:43621 - accepting connections
I20260926 01:00:34.449486 3372 mini_ranger.cc:161] Starting Ranger...
I20260926 01:00:34.463502 3372 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 01:00:34.947273 3372 mini_ranger.cc:87] Created ranger Postgres database
I20260926 01:00:34.947366 3372 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ranger-admin
2026-09-26 01:00:35,218 [I] DB FLAVOR :POSTGRES
2026-09-26 01:00:35,219 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 01:00:35,219 [I] Checking connection..
2026-09-26 01:00:35,219 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktuUsJs/build/release/bin/postgresql.jar:/tmp/dist-test-tasktuUsJs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.75.20:43621/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 01:00:35,463 [I] Checking connection passed.
2026-09-26 01:00:35,463 [I] --------- Verifying version history table ---------
2026-09-26 01:00:35,463 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktuUsJs/build/release/bin/postgresql.jar:/tmp/dist-test-tasktuUsJs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.75.20:43621/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 01:00:35,696 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 01:00:35,697 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktuUsJs/build/release/bin/postgresql.jar:/tmp/dist-test-tasktuUsJs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.75.20:43621/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 01:00:35,958 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 01:00:35,958 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 01:00:35,958 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktuUsJs/build/release/bin/postgresql.jar:/tmp/dist-test-tasktuUsJs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.75.20:43621/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasktuUsJs/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 01:00:36.200 UTC [14194] WARNING: there is no transaction in progress
2026-09-26 01:00:36,206 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 01:00:36,206 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktuUsJs/build/release/bin/postgresql.jar:/tmp/dist-test-tasktuUsJs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.75.20:43621/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 01:00:36,464 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 01:00:36,464 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 01:00:36,464 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktuUsJs/build/release/bin/postgresql.jar:/tmp/dist-test-tasktuUsJs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.75.20:43621/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 01:00:36,703 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktuUsJs/build/release/bin/postgresql.jar:/tmp/dist-test-tasktuUsJs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.75.20:43621/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 01:00:36,950 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktuUsJs/build/release/bin/postgresql.jar:/tmp/dist-test-tasktuUsJs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.75.20:43621/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-7v10.c.gcp-upstream.internal','N') ;"
2026-09-26 01:00:37,201 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktuUsJs/build/release/bin/postgresql.jar:/tmp/dist-test-tasktuUsJs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.75.20:43621/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 01:00:37,431 [I] Table x_portal_user does not exist in database ranger
2026-09-26 01:00:37,432 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktuUsJs/build/release/bin/postgresql.jar:/tmp/dist-test-tasktuUsJs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.75.20:43621/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 01:00:37,664 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 01:00:37,664 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktuUsJs/build/release/bin/postgresql.jar:/tmp/dist-test-tasktuUsJs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.75.20:43621/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 01:00:37,903 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktuUsJs/build/release/bin/postgresql.jar:/tmp/dist-test-tasktuUsJs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.75.20:43621/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 01:00:38,135 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 01:00:38,135 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktuUsJs/build/release/bin/postgresql.jar:/tmp/dist-test-tasktuUsJs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.75.20:43621/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasktuUsJs/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 01:00:39.075 UTC [14302] WARNING: there is no transaction in progress
2026-09-26 01:00:39.091 UTC [14302] WARNING: there is no transaction in progress
2026-09-26 01:00:39.106 UTC [14302] WARNING: there is no transaction in progress
2026-09-26 01:00:39.122 UTC [14302] WARNING: there is no transaction in progress
2026-09-26 01:00:39.137 UTC [14302] WARNING: there is no transaction in progress
2026-09-26 01:00:39.154 UTC [14302] WARNING: there is no transaction in progress
2026-09-26 01:00:39.207 UTC [14302] WARNING: there is no transaction in progress
2026-09-26 01:00:39.214 UTC [14302] WARNING: there is no transaction in progress
2026-09-26 01:00:39.228 UTC [14302] WARNING: there is no transaction in progress
2026-09-26 01:00:39.238 UTC [14302] WARNING: there is no transaction in progress
2026-09-26 01:00:39.247 UTC [14302] WARNING: there is no transaction in progress
2026-09-26 01:00:39.260 UTC [14302] WARNING: there is no transaction in progress
2026-09-26 01:00:39.270 UTC [14302] WARNING: there is no transaction in progress
2026-09-26 01:00:39.276 UTC [14302] WARNING: there is no transaction in progress
2026-09-26 01:00:39.283 UTC [14302] WARNING: there is no transaction in progress
2026-09-26 01:00:39.290 UTC [14302] WARNING: there is no transaction in progress
2026-09-26 01:00:39,896 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 01:00:39,896 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktuUsJs/build/release/bin/postgresql.jar:/tmp/dist-test-tasktuUsJs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.75.20:43621/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 01:00:40,142 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 01:00:40,142 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktuUsJs/build/release/bin/postgresql.jar:/tmp/dist-test-tasktuUsJs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.75.20:43621/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 01:00:40,392 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 01:00:40,393 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktuUsJs/build/release/bin/postgresql.jar:/tmp/dist-test-tasktuUsJs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.75.20:43621/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 01:00:40,619 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 01:00:40,619 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktuUsJs/build/release/bin/postgresql.jar:/tmp/dist-test-tasktuUsJs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.75.20:43621/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 01:00:40,875 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktuUsJs/build/release/bin/postgresql.jar:/tmp/dist-test-tasktuUsJs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.75.20:43621/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 01:00:41,098 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktuUsJs/build/release/bin/postgresql.jar:/tmp/dist-test-tasktuUsJs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.75.20:43621/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-7v10.c.gcp-upstream.internal';"
2026-09-26 01:00:41,341 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 01:00:41,342 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktuUsJs/build/release/bin/postgresql.jar:/tmp/dist-test-tasktuUsJs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.75.20:43621/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 01:00:41,593 [I] DB_PATCHES have already been applied
I20260926 01:00:41.598522 3372 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ranger-admin:/tmp/dist-test-tasktuUsJs/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasktuUsJs/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasktuUsJs/build/release/bin/postgresql.jar:/tmp/dist-test-tasktuUsJs/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 01:00:41.598578 3372 mini_ranger.cc:193] Using host: 127.3.75.20
I20260926 01:00:41.601567 3372 mini_ranger.cc:227] Ranger admin URL: http://127.3.75.20:59495
Sep 26, 2026 1:00:41 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 1:00:41 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 1:00:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 1:00:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 1:00:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 1:00:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 1:00:42 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 01:00:42.739918 3372 mini_ranger.cc:160] Time spent starting Ranger: real 8.290s user 0.004s sys 0.005s
Sep 26, 2026 1:01:01 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 1:01:01 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 1:01:01 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 1:01:01 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 1:01:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 1:01:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 1:01:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 1:01:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 1:01:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 1:01:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 1:01:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 1:01:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 1:01:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 1:01:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 1:01:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 1:01:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 1:01:01 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 1:01:02 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 01:01:03.389389 3372 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.75.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.75.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.75.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/krb5kdc/rangerkms_127.3.75.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.75.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/krb5kdc/rangerkms_127.3.75.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.75.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.75.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.75.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/krb5kdc/HTTP_127.3.75.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.75.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/krb5kdc/HTTP_127.3.75.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 01:01:03 dist-test-slave-7v10 krb5kdc[14089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384463, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 01:01:03.475427 3372 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 01:01:03.484807 3372 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 01:01:04.331537 3372 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 01:01:04.331626 3372 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ranger-kms
I20260926 01:01:04.331650 3372 mini_ranger_kms.cc:227] Using postgres at 127.3.75.20:43621
2026-09-26 01:01:04,510 [I] DB FLAVOR :POSTGRES
2026-09-26 01:01:04,510 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 01:01:04,511 [I] Checking connection
2026-09-26 01:01:04,511 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktuUsJs/build/release/bin/postgresql.jar:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.75.20:43621/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 01:01:04,760 [I] connection success
2026-09-26 01:01:04,760 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 01:01:04,761 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 01:01:04,761 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktuUsJs/build/release/bin/postgresql.jar:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.75.20:43621/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 01:01:05,000 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 01:01:05,001 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 01:01:05,001 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 01:01:05,001 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktuUsJs/build/release/bin/postgresql.jar:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.75.20:43621/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 01:01:05,277 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 01:01:05.721901 3372 mini_ranger_kms.cc:310] Created kms service
I20260926 01:01:05.887462 3372 mini_ranger_kms.cc:326] Created kudu user
I20260926 01:01:05.929108 3372 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 01:01:05 dist-test-slave-7v10 krb5kdc[14089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384463, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.75.20@KRBTEST.COM
I20260926 01:01:06.312556 3372 mini_ranger_kms.cc:384] Added ranger policy
I20260926 01:01:06.312888 3372 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ranger-kms:/tmp/dist-test-tasktuUsJs/build/release/bin/postgresql.jar:/tmp/dist-test-tasktuUsJs/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasktuUsJs/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 01:01:06.312923 3372 mini_ranger_kms.cc:240] Using host: 127.3.75.20
I20260926 01:01:06.316627 3372 mini_ranger_kms.cc:276] Ranger KMS PID: 14482
I20260926 01:01:06.316732 3372 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.75.20:45701
Sep 26, 2026 1:01:07 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 1:01:07 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 1:01:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 1:01:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 1:01:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 1:01:07 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 01:01:08.553032 3372 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.078s user 0.012s sys 0.149s
I20260926 01:01:08.553143 3372 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 01:01:08.553217 3372 mini_ranger_kms.cc:401] 127.3.75.20:45701/kms/v1/keys
Sep 26 01:01:08 dist-test-slave-7v10 krb5kdc[14089](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790384468, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.75.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 01:01:09 dist-test-slave-7v10 krb5kdc[14089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384468, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.75.20@KRBTEST.COM for HTTP/127.3.75.20@KRBTEST.COM
Sep 26 01:01:09 dist-test-slave-7v10 krb5kdc[14089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384468, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.75.20@KRBTEST.COM for HTTP/127.3.75.20@KRBTEST.COM
Sep 26 01:01:11 dist-test-slave-7v10 krb5kdc[14089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.75.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab.
Entry for principal kudu/127.3.75.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab.
I20260926 01:01:11.643838 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:40979
--webserver_interface=127.3.75.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:41231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.75.62:40979
--ranger_config_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.75.20:45701/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:01:11.726689 14533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:01:11.726970 14533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:01:11.727003 14533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:01:11.728243 14533 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:01:11.728286 14533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:01:11.728299 14533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:01:11.728312 14533 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:01:11.728323 14533 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:01:11.729827 14533 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:41231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.75.20:45701/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.75.62:40979
--ranger_config_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:40979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:01:11.730067 14533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:01:11.730307 14533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:01:11.736598 14538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:01:11.736619 14541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:01:11.736799 14533 server_base.cc:1034] running on GCE node
W20260926 01:01:11.736621 14539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:01:11.737145 14533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:01:11.737457 14533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:01:11.738636 14533 hybrid_clock.cc:648] HybridClock initialized: now 1790384471738624 us; error 27 us; skew 500 ppm
Sep 26 01:01:11 dist-test-slave-7v10 krb5kdc[14089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384471, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:01:11.743973 14533 init.cc:377] Logged in from keytab as kudu/127.3.75.62@KRBTEST.COM (short username kudu)
I20260926 01:01:11.744482 14533 webserver.cc:460] Webserver started at http://127.3.75.62:41655/ using document root <none> and password file <none>
I20260926 01:01:11.744710 14533 fs_manager.cc:359] Metadata directory not provided
I20260926 01:01:11.744748 14533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:01:11.744895 14533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 01:01:11 dist-test-slave-7v10 krb5kdc[14089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384471, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.62@KRBTEST.COM for HTTP/127.3.75.20@KRBTEST.COM
I20260926 01:01:11.801445 14533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/master-0/data/instance:
uuid: "945445d586324558921b40ee34bc4961"
format_stamp: "Formatted at 2026-09-26 01:01:11 on dist-test-slave-7v10"
server_key: "b9ade9e1c1528c5ea5062bdb9d368b93"
server_key_iv: "cf9b4d17656c77f2e80fa4d02b16a91e"
server_key_version: "kuduclusterkey@0"
I20260926 01:01:11.801954 14533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/master-0/wal/instance:
uuid: "945445d586324558921b40ee34bc4961"
format_stamp: "Formatted at 2026-09-26 01:01:11 on dist-test-slave-7v10"
server_key: "b9ade9e1c1528c5ea5062bdb9d368b93"
server_key_iv: "cf9b4d17656c77f2e80fa4d02b16a91e"
server_key_version: "kuduclusterkey@0"
I20260926 01:01:11.804282 14533 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260926 01:01:11.820254 14548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:01:11.820541 14533 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:01:11.820652 14533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/master-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/master-0/wal
uuid: "945445d586324558921b40ee34bc4961"
format_stamp: "Formatted at 2026-09-26 01:01:11 on dist-test-slave-7v10"
server_key: "b9ade9e1c1528c5ea5062bdb9d368b93"
server_key_iv: "cf9b4d17656c77f2e80fa4d02b16a91e"
server_key_version: "kuduclusterkey@0"
I20260926 01:01:11.820766 14533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:01:11.841135 14533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:01:11.843364 14533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:01:11.843528 14533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:01:11.849071 14533 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.62:40979
I20260926 01:01:11.849114 14600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.62:40979 every 8 connection(s)
I20260926 01:01:11.849548 14533 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
I20260926 01:01:11.850266 14601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:01:11.851002 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 14533
I20260926 01:01:11.851099 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/master-0/wal/instance
I20260926 01:01:11.851269 3372 external_mini_cluster.cc:1428] Setting key 9387c3cbeb78a6748f2c01f1b71ca1b9
I20260926 01:01:11.853242 14601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 945445d586324558921b40ee34bc4961: Bootstrap starting.
I20260926 01:01:11.853919 14601 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 945445d586324558921b40ee34bc4961: Neither blocks nor log segments found. Creating new log.
I20260926 01:01:11.854204 14601 log.cc:824] T 00000000000000000000000000000000 P 945445d586324558921b40ee34bc4961: Log is configured to *not* fsync() on all Append() calls
I20260926 01:01:11.854982 14601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 945445d586324558921b40ee34bc4961: No bootstrap required, opened a new log
I20260926 01:01:11.856995 14601 raft_consensus.cc:348] T 00000000000000000000000000000000 P 945445d586324558921b40ee34bc4961 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "945445d586324558921b40ee34bc4961" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 40979 } }
I20260926 01:01:11.857225 14601 raft_consensus.cc:374] T 00000000000000000000000000000000 P 945445d586324558921b40ee34bc4961 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:01:11.857280 14601 raft_consensus.cc:729] T 00000000000000000000000000000000 P 945445d586324558921b40ee34bc4961 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 945445d586324558921b40ee34bc4961, State: Initialized, Role: FOLLOWER
I20260926 01:01:11.857435 14601 consensus_queue.cc:260] T 00000000000000000000000000000000 P 945445d586324558921b40ee34bc4961 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "945445d586324558921b40ee34bc4961" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 40979 } }
I20260926 01:01:11.857547 14601 raft_consensus.cc:388] T 00000000000000000000000000000000 P 945445d586324558921b40ee34bc4961 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:01:11.857602 14601 raft_consensus.cc:482] T 00000000000000000000000000000000 P 945445d586324558921b40ee34bc4961 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:01:11.857676 14601 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 945445d586324558921b40ee34bc4961 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:01:11.858182 14601 raft_consensus.cc:504] T 00000000000000000000000000000000 P 945445d586324558921b40ee34bc4961 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "945445d586324558921b40ee34bc4961" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 40979 } }
I20260926 01:01:11.858309 14601 leader_election.cc:302] T 00000000000000000000000000000000 P 945445d586324558921b40ee34bc4961 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 945445d586324558921b40ee34bc4961; no voters:
I20260926 01:01:11.858490 14601 leader_election.cc:288] T 00000000000000000000000000000000 P 945445d586324558921b40ee34bc4961 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:01:11.858513 14606 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 945445d586324558921b40ee34bc4961 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:01:11.858709 14606 raft_consensus.cc:686] T 00000000000000000000000000000000 P 945445d586324558921b40ee34bc4961 [term 1 LEADER]: Becoming Leader. State: Replica: 945445d586324558921b40ee34bc4961, State: Running, Role: LEADER
I20260926 01:01:11.858829 14601 sys_catalog.cc:564] T 00000000000000000000000000000000 P 945445d586324558921b40ee34bc4961 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:01:11.858951 14606 consensus_queue.cc:237] T 00000000000000000000000000000000 P 945445d586324558921b40ee34bc4961 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "945445d586324558921b40ee34bc4961" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 40979 } }
Sep 26 01:01:11 dist-test-slave-7v10 krb5kdc[14089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 01:01:11.859316 14606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 945445d586324558921b40ee34bc4961 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "945445d586324558921b40ee34bc4961" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "945445d586324558921b40ee34bc4961" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 40979 } } }
I20260926 01:01:11.859400 14606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 945445d586324558921b40ee34bc4961 [sys.catalog]: This master's current role is: LEADER
I20260926 01:01:11.859509 14606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 945445d586324558921b40ee34bc4961 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 945445d586324558921b40ee34bc4961. Latest consensus state: current_term: 1 leader_uuid: "945445d586324558921b40ee34bc4961" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "945445d586324558921b40ee34bc4961" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 40979 } } }
I20260926 01:01:11.859561 14606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 945445d586324558921b40ee34bc4961 [sys.catalog]: This master's current role is: LEADER
I20260926 01:01:11.859661 14615 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:01:11.860180 14615 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:01:11.860409 14601 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 01:01:11.866299 14624 mvcc.cc:205] Tried to move back new op lower bound from 7333414796762214400 to 7333414796734885888. Current Snapshot: MvccSnapshot[applied={T|T < 7333414796762214400}]
I20260926 01:01:11.867136 14604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:01:11.851774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35812 (local address 127.3.75.62:40979)
0926 01:01:11.851940 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 01:01:11.851950 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 01:01:11.851991 (+ 41us) server_negotiation.cc:407] Connection header received
0926 01:01:11.852482 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:01:11.852490 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:01:11.852567 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:01:11.852720 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:01:11.853555 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:01:11.854805 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:01:11.855933 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:01:11.856600 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:01:11.856711 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:01:11.859999 (+ 3288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:01:11.860004 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:01:11.860004 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:01:11.860017 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:01:11.864790 (+ 4773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:01:11.865238 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:01:11.865242 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:01:11.865242 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:01:11.865372 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:01:11.865613 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:01:11.865614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:01:11.865614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:01:11.865943 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:01:11.866098 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 01:01:11.866652 (+ 554us) server_negotiation.cc:299] Negotiation successful
0926 01:01:11.866818 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":77,"threads_started":1}
I20260926 01:01:11.866976 14615 catalog_manager.cc:1347] Generated new cluster ID: 647b3e6574ed489e84df304528da09f6
I20260926 01:01:11.868188 14615 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:01:11.900558 14615 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:01:11.901337 14615 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:01:11.909907 14615 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 945445d586324558921b40ee34bc4961: Generated new TSK 0
I20260926 01:01:11.910102 14615 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.75.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab.
Entry for principal kudu/127.3.75.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab.
I20260926 01:01:12.553550 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.75.20:45701/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.1:0
--local_ip_for_outbound_sockets=127.3.75.1
--webserver_interface=127.3.75.1
--webserver_port=0
--tserver_master_addrs=127.3.75.62:40979
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab
--principal=kudu/127.3.75.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:41231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:01:12.658270 14646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:01:12.658631 14646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:01:12.658677 14646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:01:12.660655 14646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:01:12.660734 14646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:01:12.660778 14646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.1
I20260926 01:01:12.663179 14646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:41231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.75.20:45701/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab
--principal=kudu/127.3.75.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.75.1
--webserver_port=0
--tserver_master_addrs=127.3.75.62:40979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.1
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:01:12.663450 14646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:01:12.663762 14646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:01:12.669335 14652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:01:12.669344 14651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:01:12.669667 14646 server_base.cc:1034] running on GCE node
W20260926 01:01:12.669668 14654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:01:12.669958 14646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:01:12.670213 14646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:01:12.671406 14646 hybrid_clock.cc:648] HybridClock initialized: now 1790384472671387 us; error 33 us; skew 500 ppm
Sep 26 01:01:12 dist-test-slave-7v10 krb5kdc[14089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384472, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:01:12.674513 14646 init.cc:377] Logged in from keytab as kudu/127.3.75.1@KRBTEST.COM (short username kudu)
I20260926 01:01:12.675143 14646 webserver.cc:460] Webserver started at http://127.3.75.1:36175/ using document root <none> and password file <none>
I20260926 01:01:12.675367 14646 fs_manager.cc:359] Metadata directory not provided
I20260926 01:01:12.675434 14646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:01:12.675587 14646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 01:01:12 dist-test-slave-7v10 krb5kdc[14089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384472, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.1@KRBTEST.COM for HTTP/127.3.75.20@KRBTEST.COM
I20260926 01:01:12.690864 14646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ts-0/data/instance:
uuid: "7695a4a9cf324c1f930cbd5b5cb397be"
format_stamp: "Formatted at 2026-09-26 01:01:12 on dist-test-slave-7v10"
server_key: "746cf971a50b66fed7e6022976d9154c"
server_key_iv: "161960b69a1e621928453776abab4750"
server_key_version: "kuduclusterkey@0"
I20260926 01:01:12.691377 14646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ts-0/wal/instance:
uuid: "7695a4a9cf324c1f930cbd5b5cb397be"
format_stamp: "Formatted at 2026-09-26 01:01:12 on dist-test-slave-7v10"
server_key: "746cf971a50b66fed7e6022976d9154c"
server_key_iv: "161960b69a1e621928453776abab4750"
server_key_version: "kuduclusterkey@0"
I20260926 01:01:12.693739 14646 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 01:01:12.711162 14662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:01:12.711417 14646 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:01:12.711524 14646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ts-0/wal
uuid: "7695a4a9cf324c1f930cbd5b5cb397be"
format_stamp: "Formatted at 2026-09-26 01:01:12 on dist-test-slave-7v10"
server_key: "746cf971a50b66fed7e6022976d9154c"
server_key_iv: "161960b69a1e621928453776abab4750"
server_key_version: "kuduclusterkey@0"
I20260926 01:01:12.711629 14646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:01:12.721452 14646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:01:12.723881 14646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:01:12.724083 14646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:01:12.724383 14646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:01:12.724793 14646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:01:12.724886 14646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:01:12.724988 14646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:01:12.725072 14646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:01:12.732834 14646 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.1:43573
I20260926 01:01:12.733335 14646 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
I20260926 01:01:12.739105 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 14646
I20260926 01:01:12.739208 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ts-0/wal/instance
I20260926 01:01:12.739411 3372 external_mini_cluster.cc:1428] Setting key 5e46d35b8f214cd4fdcc28035cf33f66
I20260926 01:01:12.735136 14778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.1:43573 every 8 connection(s)
Sep 26 01:01:12 dist-test-slave-7v10 krb5kdc[14089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384472, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.1@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 01:01:12.748291 14781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:01:12.735779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:59003 (local address 127.3.75.62:40979)
0926 01:01:12.736427 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0926 01:01:12.736430 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:01:12.737322 (+ 892us) server_negotiation.cc:407] Connection header received
0926 01:01:12.737353 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:01:12.737354 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:01:12.737381 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:01:12.737449 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:01:12.738251 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:01:12.739101 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:01:12.741587 (+ 2486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:01:12.741887 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:01:12.741937 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:01:12.744754 (+ 2817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:01:12.744755 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:01:12.744755 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:01:12.744762 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:01:12.746780 (+ 2018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:01:12.747253 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:01:12.747254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:01:12.747255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:01:12.747321 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:01:12.747617 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:01:12.747618 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:01:12.747618 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:01:12.747735 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:01:12.747815 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 01:01:12.748109 (+ 294us) server_negotiation.cc:299] Negotiation successful
0926 01:01:12.748194 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":81,"threads_started":1}
I20260926 01:01:12.748490 14782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:01:12.736122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:40979 (local address 127.3.75.1:59003)
0926 01:01:12.736488 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0926 01:01:12.736531 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 01:01:12.737173 (+ 642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:01:12.737593 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:01:12.737598 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:01:12.737848 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:01:12.738135 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:01:12.738142 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:01:12.739335 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:01:12.739336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:01:12.741476 (+ 2140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:01:12.741479 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:01:12.742033 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:01:12.742034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:01:12.742181 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:01:12.742882 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:01:12.742910 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:01:12.744634 (+ 1724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:01:12.746913 (+ 2279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:01:12.746920 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:01:12.746923 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:01:12.747155 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:01:12.747403 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:01:12.747404 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:01:12.747405 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:01:12.747513 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:01:12.747806 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:01:12.747813 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:01:12.747969 (+ 156us) client_negotiation.cc:769] Sending connection context
0926 01:01:12.748078 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 01:01:12.748225 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":87,"threads_started":1}
I20260926 01:01:12.749004 14779 heartbeater.cc:347] Connected to a master server at 127.3.75.62:40979
I20260926 01:01:12.749122 14779 heartbeater.cc:464] Registering TS with master...
I20260926 01:01:12.749305 14779 heartbeater.cc:511] Master 127.3.75.62:40979 requested a full tablet report, sending...
I20260926 01:01:12.749823 14565 ts_manager.cc:194] Registered new tserver with Master: 7695a4a9cf324c1f930cbd5b5cb397be (127.3.75.1:43573)
I20260926 01:01:12.750517 14565 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.75.1@KRBTEST.COM'} at 127.3.75.1:59003
WARNING: no policy specified for kudu/127.3.75.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab.
Entry for principal kudu/127.3.75.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab.
I20260926 01:01:12.773629 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.75.20:45701/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.2:0
--local_ip_for_outbound_sockets=127.3.75.2
--webserver_interface=127.3.75.2
--webserver_port=0
--tserver_master_addrs=127.3.75.62:40979
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab
--principal=kudu/127.3.75.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:41231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 01:01:12 dist-test-slave-7v10 krb5kdc[14089](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790384472, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 01:01:12.888749 14785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:01:12.889045 14785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:01:12.889083 14785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:01:12.890825 14785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:01:12.890904 14785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:01:12.890949 14785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.2
I20260926 01:01:12.893078 14785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:41231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.75.20:45701/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab
--principal=kudu/127.3.75.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.75.2
--webserver_port=0
--tserver_master_addrs=127.3.75.62:40979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.2
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:01:12.893321 14785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:01:12.893652 14785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:01:12.902834 14796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:01:12.903282 14792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:01:12.903976 14793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:01:12.904737 14785 server_base.cc:1034] running on GCE node
I20260926 01:01:12.904950 14785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:01:12.905218 14785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:01:12.906396 14785 hybrid_clock.cc:648] HybridClock initialized: now 1790384472906354 us; error 56 us; skew 500 ppm
Sep 26 01:01:12 dist-test-slave-7v10 krb5kdc[14089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384472, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:01:12.910123 14785 init.cc:377] Logged in from keytab as kudu/127.3.75.2@KRBTEST.COM (short username kudu)
I20260926 01:01:12.910678 14785 webserver.cc:460] Webserver started at http://127.3.75.2:39875/ using document root <none> and password file <none>
I20260926 01:01:12.910995 14785 fs_manager.cc:359] Metadata directory not provided
I20260926 01:01:12.911056 14785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:01:12.911194 14785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 01:01:12 dist-test-slave-7v10 krb5kdc[14089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384472, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.2@KRBTEST.COM for HTTP/127.3.75.20@KRBTEST.COM
I20260926 01:01:12.928622 14785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ts-1/data/instance:
uuid: "2e8fc58c67c44a4fb093683fcb7009a3"
format_stamp: "Formatted at 2026-09-26 01:01:12 on dist-test-slave-7v10"
server_key: "5d0d1071d70e299efa1bb977b0eb7e81"
server_key_iv: "552c7dac4e14ec7ede708fa252c8e9a3"
server_key_version: "kuduclusterkey@0"
I20260926 01:01:12.929077 14785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ts-1/wal/instance:
uuid: "2e8fc58c67c44a4fb093683fcb7009a3"
format_stamp: "Formatted at 2026-09-26 01:01:12 on dist-test-slave-7v10"
server_key: "5d0d1071d70e299efa1bb977b0eb7e81"
server_key_iv: "552c7dac4e14ec7ede708fa252c8e9a3"
server_key_version: "kuduclusterkey@0"
I20260926 01:01:12.932011 14785 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260926 01:01:12.946957 14803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:01:12.947257 14785 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260926 01:01:12.947374 14785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ts-1/wal
uuid: "2e8fc58c67c44a4fb093683fcb7009a3"
format_stamp: "Formatted at 2026-09-26 01:01:12 on dist-test-slave-7v10"
server_key: "5d0d1071d70e299efa1bb977b0eb7e81"
server_key_iv: "552c7dac4e14ec7ede708fa252c8e9a3"
server_key_version: "kuduclusterkey@0"
I20260926 01:01:12.947474 14785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:01:12.964233 14785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:01:12.966629 14785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:01:12.966820 14785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:01:12.967123 14785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:01:12.967540 14785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:01:12.967629 14785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:01:12.967701 14785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:01:12.967759 14785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:01:12.975595 14785 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.2:39331
I20260926 01:01:12.976081 14785 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
I20260926 01:01:12.979369 14916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.2:39331 every 8 connection(s)
I20260926 01:01:12.982452 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 14785
I20260926 01:01:12.982558 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ts-1/wal/instance
I20260926 01:01:12.982750 3372 external_mini_cluster.cc:1428] Setting key 77273a5bfd2403b4d031935d9ac154ab
Sep 26 01:01:12 dist-test-slave-7v10 krb5kdc[14089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384472, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.2@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 01:01:12.989948 14781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:01:12.979577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:57623 (local address 127.3.75.62:40979)
0926 01:01:12.979626 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 01:01:12.979628 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:01:12.979735 (+ 107us) server_negotiation.cc:407] Connection header received
0926 01:01:12.980452 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:01:12.980454 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:01:12.980466 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:01:12.980538 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:01:12.981261 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:01:12.982050 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:01:12.983539 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:01:12.983837 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:01:12.983885 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:01:12.986531 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:01:12.986532 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:01:12.986533 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:01:12.986539 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:01:12.988330 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:01:12.988833 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:01:12.988834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:01:12.988834 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:01:12.988910 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:01:12.989190 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:01:12.989191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:01:12.989191 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:01:12.989320 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:01:12.989363 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 01:01:12.989765 (+ 402us) server_negotiation.cc:299] Negotiation successful
0926 01:01:12.989854 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 01:01:12.990863 14920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:01:12.979285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:40979 (local address 127.3.75.2:57623)
0926 01:01:12.979583 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0926 01:01:12.979626 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 01:01:12.980303 (+ 677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:01:12.980635 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:01:12.980639 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:01:12.980867 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:01:12.981140 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:01:12.981146 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:01:12.982163 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:01:12.982164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:01:12.983446 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:01:12.983449 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:01:12.983978 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:01:12.983978 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:01:12.984118 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:01:12.984740 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:01:12.984749 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:01:12.986426 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:01:12.988442 (+ 2016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:01:12.988448 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:01:12.988450 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:01:12.988732 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:01:12.988994 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:01:12.988995 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:01:12.988995 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:01:12.989119 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:01:12.989424 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:01:12.989430 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:01:12.989604 (+ 174us) client_negotiation.cc:769] Sending connection context
0926 01:01:12.990472 (+ 868us) client_negotiation.cc:240] Negotiation successful
0926 01:01:12.990630 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":83,"threads_started":1}
I20260926 01:01:12.991525 14918 heartbeater.cc:347] Connected to a master server at 127.3.75.62:40979
I20260926 01:01:12.991608 14918 heartbeater.cc:464] Registering TS with master...
I20260926 01:01:12.991780 14918 heartbeater.cc:511] Master 127.3.75.62:40979 requested a full tablet report, sending...
I20260926 01:01:12.992209 14565 ts_manager.cc:194] Registered new tserver with Master: 2e8fc58c67c44a4fb093683fcb7009a3 (127.3.75.2:39331)
I20260926 01:01:12.992666 14565 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.75.2@KRBTEST.COM'} at 127.3.75.2:57623
WARNING: no policy specified for kudu/127.3.75.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab.
Entry for principal kudu/127.3.75.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab.
I20260926 01:01:13.025854 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.75.20:45701/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.3:0
--local_ip_for_outbound_sockets=127.3.75.3
--webserver_interface=127.3.75.3
--webserver_port=0
--tserver_master_addrs=127.3.75.62:40979
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab
--principal=kudu/127.3.75.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:41231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:01:13.148664 14923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:01:13.149050 14923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:01:13.149150 14923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:01:13.150897 14923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:01:13.151000 14923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:01:13.151089 14923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.3
I20260926 01:01:13.153605 14923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:41231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.3.75.20:45701/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab
--principal=kudu/127.3.75.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.75.3
--webserver_port=0
--tserver_master_addrs=127.3.75.62:40979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.3
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:01:13.154111 14923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:01:13.154464 14923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:01:13.161619 14929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:01:13.162778 14923 server_base.cc:1034] running on GCE node
W20260926 01:01:13.167024 14928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:01:13.167232 14931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:01:13.167591 14923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:01:13.167892 14923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:01:13.171165 14923 hybrid_clock.cc:648] HybridClock initialized: now 1790384473171128 us; error 48 us; skew 500 ppm
Sep 26 01:01:13 dist-test-slave-7v10 krb5kdc[14089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384473, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:01:13.174443 14923 init.cc:377] Logged in from keytab as kudu/127.3.75.3@KRBTEST.COM (short username kudu)
I20260926 01:01:13.175083 14923 webserver.cc:460] Webserver started at http://127.3.75.3:36971/ using document root <none> and password file <none>
I20260926 01:01:13.175360 14923 fs_manager.cc:359] Metadata directory not provided
I20260926 01:01:13.175419 14923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:01:13.175542 14923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 01:01:13 dist-test-slave-7v10 krb5kdc[14089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384473, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.3@KRBTEST.COM for HTTP/127.3.75.20@KRBTEST.COM
I20260926 01:01:13.189251 14923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ts-2/data/instance:
uuid: "8a2911e3464044b495e22fedceecf94a"
format_stamp: "Formatted at 2026-09-26 01:01:13 on dist-test-slave-7v10"
server_key: "7f604b2eb2dc828c5dfc756020edf770"
server_key_iv: "9122e5ea126cc9db27c4642dcdb7d870"
server_key_version: "kuduclusterkey@0"
I20260926 01:01:13.190953 14923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ts-2/wal/instance:
uuid: "8a2911e3464044b495e22fedceecf94a"
format_stamp: "Formatted at 2026-09-26 01:01:13 on dist-test-slave-7v10"
server_key: "7f604b2eb2dc828c5dfc756020edf770"
server_key_iv: "9122e5ea126cc9db27c4642dcdb7d870"
server_key_version: "kuduclusterkey@0"
I20260926 01:01:13.199021 14923 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.000s sys 0.001s
I20260926 01:01:13.215065 14938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:01:13.215272 14923 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 01:01:13.215366 14923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ts-2/wal
uuid: "8a2911e3464044b495e22fedceecf94a"
format_stamp: "Formatted at 2026-09-26 01:01:13 on dist-test-slave-7v10"
server_key: "7f604b2eb2dc828c5dfc756020edf770"
server_key_iv: "9122e5ea126cc9db27c4642dcdb7d870"
server_key_version: "kuduclusterkey@0"
I20260926 01:01:13.215471 14923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:01:13.240137 14923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:01:13.243007 14923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:01:13.243191 14923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:01:13.243460 14923 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:01:13.243829 14923 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:01:13.243920 14923 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:01:13.248167 14923 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:01:13.248211 14923 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:01:13.257138 14923 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.3:44709
I20260926 01:01:13.257652 14923 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
I20260926 01:01:13.258055 15051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.3:44709 every 8 connection(s)
I20260926 01:01:13.262259 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 14923
I20260926 01:01:13.262357 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ts-2/wal/instance
I20260926 01:01:13.262517 3372 external_mini_cluster.cc:1428] Setting key 554a610498f6a8a677d65f4a0ac7dd5a
Sep 26 01:01:13 dist-test-slave-7v10 krb5kdc[14089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384473, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.3@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 01:01:13.273473 14781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:01:13.259404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.3:48753 (local address 127.3.75.62:40979)
0926 01:01:13.259446 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 01:01:13.259448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:01:13.259458 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:01:13.260190 (+ 732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:01:13.260191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:01:13.260204 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:01:13.260234 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:01:13.261220 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:01:13.262013 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:01:13.267073 (+ 5060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:01:13.267383 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:01:13.267438 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:01:13.270093 (+ 2655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:01:13.270095 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:01:13.270095 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:01:13.270102 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:01:13.271870 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:01:13.272341 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:01:13.272343 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:01:13.272343 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:01:13.272417 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:01:13.272709 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:01:13.272710 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:01:13.272710 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:01:13.272838 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:01:13.272880 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 01:01:13.273275 (+ 395us) server_negotiation.cc:299] Negotiation successful
0926 01:01:13.273381 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 01:01:13.274272 15054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:01:13.259092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:40979 (local address 127.3.75.3:48753)
0926 01:01:13.259303 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 01:01:13.259365 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 01:01:13.260038 (+ 673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:01:13.260419 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:01:13.260423 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:01:13.260673 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:01:13.260956 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:01:13.260962 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:01:13.265491 (+ 4529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:01:13.265492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:01:13.266951 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:01:13.266954 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:01:13.267535 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:01:13.267536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:01:13.267678 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:01:13.268298 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:01:13.268308 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:01:13.269972 (+ 1664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:01:13.271964 (+ 1992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:01:13.271970 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:01:13.271972 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:01:13.272250 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:01:13.272494 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:01:13.272495 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:01:13.272496 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:01:13.272631 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:01:13.272925 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:01:13.272931 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:01:13.273117 (+ 186us) client_negotiation.cc:769] Sending connection context
0926 01:01:13.273844 (+ 727us) client_negotiation.cc:240] Negotiation successful
0926 01:01:13.274028 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":81,"threads_started":1}
I20260926 01:01:13.274981 15052 heartbeater.cc:347] Connected to a master server at 127.3.75.62:40979
I20260926 01:01:13.275074 15052 heartbeater.cc:464] Registering TS with master...
I20260926 01:01:13.275250 15052 heartbeater.cc:511] Master 127.3.75.62:40979 requested a full tablet report, sending...
I20260926 01:01:13.275660 14565 ts_manager.cc:194] Registered new tserver with Master: 8a2911e3464044b495e22fedceecf94a (127.3.75.3:44709)
I20260926 01:01:13.276155 14565 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.75.3@KRBTEST.COM'} at 127.3.75.3:48753
I20260926 01:01:13.284638 3372 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 01:01:13.296288 14781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:01:13.286652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35820 (local address 127.3.75.62:40979)
0926 01:01:13.286690 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 01:01:13.286692 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:01:13.287049 (+ 357us) server_negotiation.cc:407] Connection header received
0926 01:01:13.287177 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:01:13.287177 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:01:13.287187 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:01:13.287216 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:01:13.288629 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:01:13.289339 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:01:13.291068 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:01:13.291370 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:01:13.291416 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:01:13.293037 (+ 1621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:01:13.293038 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:01:13.293038 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:01:13.293046 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:01:13.294800 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:01:13.295244 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:01:13.295245 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:01:13.295245 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:01:13.295304 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:01:13.295539 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:01:13.295540 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:01:13.295540 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:01:13.295686 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:01:13.295773 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 01:01:13.296104 (+ 331us) server_negotiation.cc:299] Negotiation successful
0926 01:01:13.296194 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 01:01:13.298535 14565 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35820:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:01:13.300938 14565 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:01:13.319278 15067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:01:13.304629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.1:43573 (local address 127.0.0.1:54846)
0926 01:01:13.306078 (+ 1449us) negotiation.cc:107] Waiting for socket to connect
0926 01:01:13.306086 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 01:01:13.306121 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:01:13.310759 (+ 4638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:01:13.310763 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:01:13.310765 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:01:13.310830 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:01:13.310831 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:01:13.312755 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:01:13.312756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:01:13.314097 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:01:13.314100 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:01:13.318968 (+ 4868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:01:13.318969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:01:13.319055 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:01:13.319071 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 01:01:13.319140 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 01:01:13.319191 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1434,"thread_start_us":63,"threads_started":1}
I20260926 01:01:13.320444 15066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:01:13.304411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.2:39331 (local address 127.0.0.1:60092)
0926 01:01:13.305857 (+ 1446us) negotiation.cc:107] Waiting for socket to connect
0926 01:01:13.305898 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 01:01:13.305961 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:01:13.311041 (+ 5080us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:01:13.311042 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:01:13.311044 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:01:13.311082 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:01:13.311083 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:01:13.315905 (+ 4822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:01:13.315906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:01:13.317109 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:01:13.317112 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:01:13.320200 (+ 3088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:01:13.320201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:01:13.320268 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:01:13.320272 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:01:13.320324 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 01:01:13.320372 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1417,"thread_start_us":68,"threads_started":1}
I20260926 01:01:13.321012 15070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:01:13.305446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54846 (local address 127.3.75.1:43573)
0926 01:01:13.310525 (+ 5079us) server_negotiation.cc:206] Beginning negotiation
0926 01:01:13.310530 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:01:13.310540 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:01:13.310573 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:01:13.310575 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:01:13.310590 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:01:13.310666 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:01:13.311177 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:01:13.312318 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:01:13.317213 (+ 4895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:01:13.317965 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:01:13.318037 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:01:13.318095 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 01:01:13.320885 (+ 2790us) server_negotiation.cc:299] Negotiation successful
0926 01:01:13.320937 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5045,"thread_start_us":59,"threads_started":1}
I20260926 01:01:13.321550 15069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:01:13.305211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60092 (local address 127.3.75.2:39331)
0926 01:01:13.306230 (+ 1019us) server_negotiation.cc:206] Beginning negotiation
0926 01:01:13.306235 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:01:13.306245 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:01:13.306284 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:01:13.306286 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:01:13.306302 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:01:13.310959 (+ 4657us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:01:13.312436 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:01:13.315827 (+ 3391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:01:13.318149 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:01:13.319975 (+ 1826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:01:13.320042 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:01:13.320163 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 01:01:13.321425 (+ 1262us) server_negotiation.cc:299] Negotiation successful
0926 01:01:13.321467 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":984,"thread_start_us":87,"threads_started":1}
I20260926 01:01:13.321892 14712 tablet_service.cc:1467] Processing CreateTablet for tablet df0fe28e66344ebdb74d7a08e3d99347 (DEFAULT_TABLE table=test-table [id=ab997f2d454049a6bc4c75f8c2d95e74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:01:13.322304 14712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df0fe28e66344ebdb74d7a08e3d99347. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:01:13.322353 14851 tablet_service.cc:1467] Processing CreateTablet for tablet df0fe28e66344ebdb74d7a08e3d99347 (DEFAULT_TABLE table=test-table [id=ab997f2d454049a6bc4c75f8c2d95e74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:01:13.322666 14851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df0fe28e66344ebdb74d7a08e3d99347. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:01:13.323664 15068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:01:13.305207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39304 (local address 127.3.75.3:44709)
0926 01:01:13.314982 (+ 9775us) server_negotiation.cc:206] Beginning negotiation
0926 01:01:13.314986 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:01:13.315675 (+ 689us) server_negotiation.cc:407] Connection header received
0926 01:01:13.315756 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:01:13.315758 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:01:13.315773 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:01:13.315847 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:01:13.316049 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:01:13.317116 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:01:13.322052 (+ 4936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:01:13.322797 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:01:13.323170 (+ 373us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:01:13.323274 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 01:01:13.323520 (+ 246us) server_negotiation.cc:299] Negotiation successful
0926 01:01:13.323564 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9734,"thread_start_us":107,"threads_started":1}
I20260926 01:01:13.323899 15065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:01:13.304799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.3:44709 (local address 127.0.0.1:39304)
0926 01:01:13.315577 (+ 10778us) negotiation.cc:107] Waiting for socket to connect
0926 01:01:13.315587 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:01:13.315677 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:01:13.315941 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:01:13.315942 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:01:13.315943 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:01:13.315975 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:01:13.315976 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:01:13.317213 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:01:13.317214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:01:13.318267 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:01:13.318269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:01:13.323337 (+ 5068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:01:13.323338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:01:13.323431 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:01:13.323435 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:01:13.323781 (+ 346us) client_negotiation.cc:240] Negotiation successful
0926 01:01:13.323827 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10761,"thread_start_us":77,"threads_started":1}
I20260926 01:01:13.324553 14986 tablet_service.cc:1467] Processing CreateTablet for tablet df0fe28e66344ebdb74d7a08e3d99347 (DEFAULT_TABLE table=test-table [id=ab997f2d454049a6bc4c75f8c2d95e74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:01:13.324873 14986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df0fe28e66344ebdb74d7a08e3d99347. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:01:13.329272 15071 tablet_bootstrap.cc:492] T df0fe28e66344ebdb74d7a08e3d99347 P 7695a4a9cf324c1f930cbd5b5cb397be: Bootstrap starting.
I20260926 01:01:13.329994 15071 tablet_bootstrap.cc:654] T df0fe28e66344ebdb74d7a08e3d99347 P 7695a4a9cf324c1f930cbd5b5cb397be: Neither blocks nor log segments found. Creating new log.
I20260926 01:01:13.330379 15071 log.cc:824] T df0fe28e66344ebdb74d7a08e3d99347 P 7695a4a9cf324c1f930cbd5b5cb397be: Log is configured to *not* fsync() on all Append() calls
I20260926 01:01:13.330775 15072 tablet_bootstrap.cc:492] T df0fe28e66344ebdb74d7a08e3d99347 P 2e8fc58c67c44a4fb093683fcb7009a3: Bootstrap starting.
I20260926 01:01:13.331851 15072 tablet_bootstrap.cc:654] T df0fe28e66344ebdb74d7a08e3d99347 P 2e8fc58c67c44a4fb093683fcb7009a3: Neither blocks nor log segments found. Creating new log.
I20260926 01:01:13.332198 15072 log.cc:824] T df0fe28e66344ebdb74d7a08e3d99347 P 2e8fc58c67c44a4fb093683fcb7009a3: Log is configured to *not* fsync() on all Append() calls
I20260926 01:01:13.332607 15073 tablet_bootstrap.cc:492] T df0fe28e66344ebdb74d7a08e3d99347 P 8a2911e3464044b495e22fedceecf94a: Bootstrap starting.
I20260926 01:01:13.333604 15073 tablet_bootstrap.cc:654] T df0fe28e66344ebdb74d7a08e3d99347 P 8a2911e3464044b495e22fedceecf94a: Neither blocks nor log segments found. Creating new log.
I20260926 01:01:13.333945 15073 log.cc:824] T df0fe28e66344ebdb74d7a08e3d99347 P 8a2911e3464044b495e22fedceecf94a: Log is configured to *not* fsync() on all Append() calls
I20260926 01:01:13.335166 15071 tablet_bootstrap.cc:492] T df0fe28e66344ebdb74d7a08e3d99347 P 7695a4a9cf324c1f930cbd5b5cb397be: No bootstrap required, opened a new log
I20260926 01:01:13.335301 15071 ts_tablet_manager.cc:1397] T df0fe28e66344ebdb74d7a08e3d99347 P 7695a4a9cf324c1f930cbd5b5cb397be: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260926 01:01:13.337917 15071 raft_consensus.cc:348] T df0fe28e66344ebdb74d7a08e3d99347 P 7695a4a9cf324c1f930cbd5b5cb397be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e8fc58c67c44a4fb093683fcb7009a3" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39331 } } peers { permanent_uuid: "8a2911e3464044b495e22fedceecf94a" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 44709 } } peers { permanent_uuid: "7695a4a9cf324c1f930cbd5b5cb397be" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 43573 } }
I20260926 01:01:13.338192 15071 raft_consensus.cc:374] T df0fe28e66344ebdb74d7a08e3d99347 P 7695a4a9cf324c1f930cbd5b5cb397be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:01:13.338279 15071 raft_consensus.cc:729] T df0fe28e66344ebdb74d7a08e3d99347 P 7695a4a9cf324c1f930cbd5b5cb397be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7695a4a9cf324c1f930cbd5b5cb397be, State: Initialized, Role: FOLLOWER
I20260926 01:01:13.338481 15071 consensus_queue.cc:260] T df0fe28e66344ebdb74d7a08e3d99347 P 7695a4a9cf324c1f930cbd5b5cb397be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e8fc58c67c44a4fb093683fcb7009a3" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39331 } } peers { permanent_uuid: "8a2911e3464044b495e22fedceecf94a" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 44709 } } peers { permanent_uuid: "7695a4a9cf324c1f930cbd5b5cb397be" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 43573 } }
I20260926 01:01:13.338832 15071 ts_tablet_manager.cc:1428] T df0fe28e66344ebdb74d7a08e3d99347 P 7695a4a9cf324c1f930cbd5b5cb397be: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 01:01:13.339006 14779 heartbeater.cc:503] Master 127.3.75.62:40979 was elected leader, sending a full tablet report...
I20260926 01:01:13.339834 15072 tablet_bootstrap.cc:492] T df0fe28e66344ebdb74d7a08e3d99347 P 2e8fc58c67c44a4fb093683fcb7009a3: No bootstrap required, opened a new log
I20260926 01:01:13.339903 15072 ts_tablet_manager.cc:1397] T df0fe28e66344ebdb74d7a08e3d99347 P 2e8fc58c67c44a4fb093683fcb7009a3: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20260926 01:01:13.341873 15072 raft_consensus.cc:348] T df0fe28e66344ebdb74d7a08e3d99347 P 2e8fc58c67c44a4fb093683fcb7009a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e8fc58c67c44a4fb093683fcb7009a3" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39331 } } peers { permanent_uuid: "8a2911e3464044b495e22fedceecf94a" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 44709 } } peers { permanent_uuid: "7695a4a9cf324c1f930cbd5b5cb397be" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 43573 } }
I20260926 01:01:13.342023 15072 raft_consensus.cc:374] T df0fe28e66344ebdb74d7a08e3d99347 P 2e8fc58c67c44a4fb093683fcb7009a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:01:13.342072 15072 raft_consensus.cc:729] T df0fe28e66344ebdb74d7a08e3d99347 P 2e8fc58c67c44a4fb093683fcb7009a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e8fc58c67c44a4fb093683fcb7009a3, State: Initialized, Role: FOLLOWER
I20260926 01:01:13.342198 15072 consensus_queue.cc:260] T df0fe28e66344ebdb74d7a08e3d99347 P 2e8fc58c67c44a4fb093683fcb7009a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e8fc58c67c44a4fb093683fcb7009a3" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39331 } } peers { permanent_uuid: "8a2911e3464044b495e22fedceecf94a" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 44709 } } peers { permanent_uuid: "7695a4a9cf324c1f930cbd5b5cb397be" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 43573 } }
I20260926 01:01:13.342444 15072 ts_tablet_manager.cc:1428] T df0fe28e66344ebdb74d7a08e3d99347 P 2e8fc58c67c44a4fb093683fcb7009a3: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260926 01:01:13.342662 14918 heartbeater.cc:503] Master 127.3.75.62:40979 was elected leader, sending a full tablet report...
I20260926 01:01:13.343871 15073 tablet_bootstrap.cc:492] T df0fe28e66344ebdb74d7a08e3d99347 P 8a2911e3464044b495e22fedceecf94a: No bootstrap required, opened a new log
I20260926 01:01:13.343942 15073 ts_tablet_manager.cc:1397] T df0fe28e66344ebdb74d7a08e3d99347 P 8a2911e3464044b495e22fedceecf94a: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20260926 01:01:13.346156 15073 raft_consensus.cc:348] T df0fe28e66344ebdb74d7a08e3d99347 P 8a2911e3464044b495e22fedceecf94a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e8fc58c67c44a4fb093683fcb7009a3" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39331 } } peers { permanent_uuid: "8a2911e3464044b495e22fedceecf94a" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 44709 } } peers { permanent_uuid: "7695a4a9cf324c1f930cbd5b5cb397be" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 43573 } }
I20260926 01:01:13.346380 15073 raft_consensus.cc:374] T df0fe28e66344ebdb74d7a08e3d99347 P 8a2911e3464044b495e22fedceecf94a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:01:13.346498 15073 raft_consensus.cc:729] T df0fe28e66344ebdb74d7a08e3d99347 P 8a2911e3464044b495e22fedceecf94a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a2911e3464044b495e22fedceecf94a, State: Initialized, Role: FOLLOWER
I20260926 01:01:13.346683 15073 consensus_queue.cc:260] T df0fe28e66344ebdb74d7a08e3d99347 P 8a2911e3464044b495e22fedceecf94a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e8fc58c67c44a4fb093683fcb7009a3" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39331 } } peers { permanent_uuid: "8a2911e3464044b495e22fedceecf94a" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 44709 } } peers { permanent_uuid: "7695a4a9cf324c1f930cbd5b5cb397be" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 43573 } }
I20260926 01:01:13.347025 15073 ts_tablet_manager.cc:1428] T df0fe28e66344ebdb74d7a08e3d99347 P 8a2911e3464044b495e22fedceecf94a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 01:01:13.347177 15052 heartbeater.cc:503] Master 127.3.75.62:40979 was elected leader, sending a full tablet report...
Sep 26 01:01:13 dist-test-slave-7v10 krb5kdc[14089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384472, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.62@KRBTEST.COM for HTTP/127.3.75.20@KRBTEST.COM
W20260926 01:01:13.481686 14919 tablet.cc:2369] T df0fe28e66344ebdb74d7a08e3d99347 P 2e8fc58c67c44a4fb093683fcb7009a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 01:01:13.485503 14780 tablet.cc:2369] T df0fe28e66344ebdb74d7a08e3d99347 P 7695a4a9cf324c1f930cbd5b5cb397be: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 01:01:13.508692 15053 tablet.cc:2369] T df0fe28e66344ebdb74d7a08e3d99347 P 8a2911e3464044b495e22fedceecf94a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:01:13.616951 15077 raft_consensus.cc:482] T df0fe28e66344ebdb74d7a08e3d99347 P 7695a4a9cf324c1f930cbd5b5cb397be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:01:13.617125 15077 raft_consensus.cc:504] T df0fe28e66344ebdb74d7a08e3d99347 P 7695a4a9cf324c1f930cbd5b5cb397be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e8fc58c67c44a4fb093683fcb7009a3" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39331 } } peers { permanent_uuid: "8a2911e3464044b495e22fedceecf94a" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 44709 } } peers { permanent_uuid: "7695a4a9cf324c1f930cbd5b5cb397be" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 43573 } }
I20260926 01:01:13.617509 15077 leader_election.cc:288] T df0fe28e66344ebdb74d7a08e3d99347 P 7695a4a9cf324c1f930cbd5b5cb397be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2e8fc58c67c44a4fb093683fcb7009a3 (127.3.75.2:39331), 8a2911e3464044b495e22fedceecf94a (127.3.75.3:44709)
I20260926 01:01:13.621941 15068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:01:13.618003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:60193 (local address 127.3.75.3:44709)
0926 01:01:13.618059 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 01:01:13.618061 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:01:13.618156 (+ 95us) server_negotiation.cc:407] Connection header received
0926 01:01:13.618271 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:01:13.618272 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:01:13.618283 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:01:13.618315 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:01:13.618584 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:01:13.619314 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:01:13.620915 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:01:13.621495 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:01:13.621544 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:01:13.621555 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 01:01:13.621825 (+ 270us) server_negotiation.cc:299] Negotiation successful
0926 01:01:13.621871 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 01:01:13.621950 15069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:01:13.618100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:57959 (local address 127.3.75.2:39331)
0926 01:01:13.618139 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 01:01:13.618140 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:01:13.618262 (+ 122us) server_negotiation.cc:407] Connection header received
0926 01:01:13.618383 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:01:13.618384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:01:13.618396 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:01:13.618424 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:01:13.618628 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:01:13.619360 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:01:13.620928 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:01:13.621527 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:01:13.621569 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:01:13.621579 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 01:01:13.621844 (+ 265us) server_negotiation.cc:299] Negotiation successful
0926 01:01:13.621880 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 01:01:13.622203 15081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:01:13.617779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.3:44709 (local address 127.3.75.1:60193)
0926 01:01:13.618083 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0926 01:01:13.618095 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:01:13.618187 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:01:13.618399 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:01:13.618400 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:01:13.618402 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:01:13.618500 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:01:13.618502 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:01:13.619442 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:01:13.619443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:01:13.620805 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:01:13.620807 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:01:13.621660 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:01:13.621661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:01:13.621736 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:01:13.621740 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:01:13.622065 (+ 325us) client_negotiation.cc:240] Negotiation successful
0926 01:01:13.622107 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"mutex_wait_us":26,"thread_start_us":107,"threads_started":1}
I20260926 01:01:13.622597 15005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df0fe28e66344ebdb74d7a08e3d99347" candidate_uuid: "7695a4a9cf324c1f930cbd5b5cb397be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a2911e3464044b495e22fedceecf94a" is_pre_election: true
I20260926 01:01:13.622758 15005 raft_consensus.cc:2454] T df0fe28e66344ebdb74d7a08e3d99347 P 8a2911e3464044b495e22fedceecf94a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7695a4a9cf324c1f930cbd5b5cb397be in term 0.
I20260926 01:01:13.623003 14666 leader_election.cc:302] T df0fe28e66344ebdb74d7a08e3d99347 P 7695a4a9cf324c1f930cbd5b5cb397be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7695a4a9cf324c1f930cbd5b5cb397be, 8a2911e3464044b495e22fedceecf94a; no voters:
I20260926 01:01:13.623121 15077 raft_consensus.cc:2781] T df0fe28e66344ebdb74d7a08e3d99347 P 7695a4a9cf324c1f930cbd5b5cb397be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:01:13.623157 15077 raft_consensus.cc:482] T df0fe28e66344ebdb74d7a08e3d99347 P 7695a4a9cf324c1f930cbd5b5cb397be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:01:13.623176 15077 raft_consensus.cc:3037] T df0fe28e66344ebdb74d7a08e3d99347 P 7695a4a9cf324c1f930cbd5b5cb397be [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:01:13.624017 15077 raft_consensus.cc:504] T df0fe28e66344ebdb74d7a08e3d99347 P 7695a4a9cf324c1f930cbd5b5cb397be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e8fc58c67c44a4fb093683fcb7009a3" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39331 } } peers { permanent_uuid: "8a2911e3464044b495e22fedceecf94a" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 44709 } } peers { permanent_uuid: "7695a4a9cf324c1f930cbd5b5cb397be" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 43573 } }
I20260926 01:01:13.624135 15077 leader_election.cc:288] T df0fe28e66344ebdb74d7a08e3d99347 P 7695a4a9cf324c1f930cbd5b5cb397be [CANDIDATE]: Term 1 election: Requested vote from peers 2e8fc58c67c44a4fb093683fcb7009a3 (127.3.75.2:39331), 8a2911e3464044b495e22fedceecf94a (127.3.75.3:44709)
I20260926 01:01:13.624536 15005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df0fe28e66344ebdb74d7a08e3d99347" candidate_uuid: "7695a4a9cf324c1f930cbd5b5cb397be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a2911e3464044b495e22fedceecf94a"
I20260926 01:01:13.624608 15005 raft_consensus.cc:3037] T df0fe28e66344ebdb74d7a08e3d99347 P 8a2911e3464044b495e22fedceecf94a [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:01:13.625444 15005 raft_consensus.cc:2454] T df0fe28e66344ebdb74d7a08e3d99347 P 8a2911e3464044b495e22fedceecf94a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7695a4a9cf324c1f930cbd5b5cb397be in term 1.
I20260926 01:01:13.625664 14666 leader_election.cc:302] T df0fe28e66344ebdb74d7a08e3d99347 P 7695a4a9cf324c1f930cbd5b5cb397be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7695a4a9cf324c1f930cbd5b5cb397be, 8a2911e3464044b495e22fedceecf94a; no voters:
I20260926 01:01:13.625772 15077 raft_consensus.cc:2781] T df0fe28e66344ebdb74d7a08e3d99347 P 7695a4a9cf324c1f930cbd5b5cb397be [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:01:13.625905 15077 raft_consensus.cc:686] T df0fe28e66344ebdb74d7a08e3d99347 P 7695a4a9cf324c1f930cbd5b5cb397be [term 1 LEADER]: Becoming Leader. State: Replica: 7695a4a9cf324c1f930cbd5b5cb397be, State: Running, Role: LEADER
I20260926 01:01:13.625988 15077 consensus_queue.cc:237] T df0fe28e66344ebdb74d7a08e3d99347 P 7695a4a9cf324c1f930cbd5b5cb397be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e8fc58c67c44a4fb093683fcb7009a3" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39331 } } peers { permanent_uuid: "8a2911e3464044b495e22fedceecf94a" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 44709 } } peers { permanent_uuid: "7695a4a9cf324c1f930cbd5b5cb397be" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 43573 } }
I20260926 01:01:13.626703 15080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:01:13.617863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.2:39331 (local address 127.3.75.1:57959)
0926 01:01:13.618201 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0926 01:01:13.618211 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:01:13.618286 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:01:13.618488 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:01:13.618489 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:01:13.618491 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:01:13.618563 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:01:13.618564 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:01:13.619457 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:01:13.619458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:01:13.620821 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:01:13.620824 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:01:13.621686 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:01:13.621687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:01:13.621765 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:01:13.621769 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:01:13.622372 (+ 603us) client_negotiation.cc:240] Negotiation successful
0926 01:01:13.622417 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":68,"thread_start_us":85,"threads_started":1}
I20260926 01:01:13.626770 14565 catalog_manager.cc:5495] T df0fe28e66344ebdb74d7a08e3d99347 P 7695a4a9cf324c1f930cbd5b5cb397be reported cstate change: term changed from 0 to 1, leader changed from <none> to 7695a4a9cf324c1f930cbd5b5cb397be (127.3.75.1). New cstate: current_term: 1 leader_uuid: "7695a4a9cf324c1f930cbd5b5cb397be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e8fc58c67c44a4fb093683fcb7009a3" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a2911e3464044b495e22fedceecf94a" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 44709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7695a4a9cf324c1f930cbd5b5cb397be" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 43573 } health_report { overall_health: HEALTHY } } }
I20260926 01:01:13.627158 14871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df0fe28e66344ebdb74d7a08e3d99347" candidate_uuid: "7695a4a9cf324c1f930cbd5b5cb397be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e8fc58c67c44a4fb093683fcb7009a3" is_pre_election: true
I20260926 01:01:13.627296 14871 raft_consensus.cc:2454] T df0fe28e66344ebdb74d7a08e3d99347 P 2e8fc58c67c44a4fb093683fcb7009a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7695a4a9cf324c1f930cbd5b5cb397be in term 0.
I20260926 01:01:13.627413 14870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df0fe28e66344ebdb74d7a08e3d99347" candidate_uuid: "7695a4a9cf324c1f930cbd5b5cb397be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e8fc58c67c44a4fb093683fcb7009a3"
I20260926 01:01:13.627471 14870 raft_consensus.cc:3037] T df0fe28e66344ebdb74d7a08e3d99347 P 2e8fc58c67c44a4fb093683fcb7009a3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:01:13.628293 14870 raft_consensus.cc:2454] T df0fe28e66344ebdb74d7a08e3d99347 P 2e8fc58c67c44a4fb093683fcb7009a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7695a4a9cf324c1f930cbd5b5cb397be in term 1.
I20260926 01:01:13.665349 15070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:01:13.661841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54854 (local address 127.3.75.1:43573)
0926 01:01:13.661929 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 01:01:13.661931 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:01:13.661940 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:01:13.661990 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:01:13.661991 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:01:13.662015 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:01:13.662097 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:01:13.662302 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:01:13.663128 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:01:13.664298 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:01:13.664609 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:01:13.664678 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:01:13.664869 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:01:13.664996 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:01:13.665063 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 01:01:13.665232 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 01:01:13.665251 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 01:01:13.667553 14870 raft_consensus.cc:1261] T df0fe28e66344ebdb74d7a08e3d99347 P 2e8fc58c67c44a4fb093683fcb7009a3 [term 1 FOLLOWER]: Refusing update from remote peer 7695a4a9cf324c1f930cbd5b5cb397be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:01:13.667552 15005 raft_consensus.cc:1261] T df0fe28e66344ebdb74d7a08e3d99347 P 8a2911e3464044b495e22fedceecf94a [term 1 FOLLOWER]: Refusing update from remote peer 7695a4a9cf324c1f930cbd5b5cb397be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:01:13.667963 15077 consensus_queue.cc:1035] T df0fe28e66344ebdb74d7a08e3d99347 P 7695a4a9cf324c1f930cbd5b5cb397be [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a2911e3464044b495e22fedceecf94a" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 44709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:01:13.668370 15077 consensus_queue.cc:1035] T df0fe28e66344ebdb74d7a08e3d99347 P 7695a4a9cf324c1f930cbd5b5cb397be [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e8fc58c67c44a4fb093683fcb7009a3" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:01:13.671766 15085 mvcc.cc:205] Tried to move back new op lower bound from 7333414804139270144 to 7333414803972431872. Current Snapshot: MvccSnapshot[applied={T|T < 7333414804139270144}]
I20260926 01:01:13.674962 15086 mvcc.cc:205] Tried to move back new op lower bound from 7333414804139270144 to 7333414803972431872. Current Snapshot: MvccSnapshot[applied={T|T < 7333414804139270144 or (T in {7333414804139270144})}]
I20260926 01:01:13.675256 15089 mvcc.cc:205] Tried to move back new op lower bound from 7333414804139270144 to 7333414803972431872. Current Snapshot: MvccSnapshot[applied={T|T < 7333414804139270144 or (T in {7333414804139270144})}]
I20260926 01:01:13.676093 14565 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:01:13.676177 14565 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:01:13.676882 14565 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 945445d586324558921b40ee34bc4961: Sending DeleteTablet for 3 replicas of tablet df0fe28e66344ebdb74d7a08e3d99347
I20260926 01:01:13.677207 14986 tablet_service.cc:1514] Processing DeleteTablet for tablet df0fe28e66344ebdb74d7a08e3d99347 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:01:13 UTC) from {username='kudu', principal='kudu/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:39304
I20260926 01:01:13.677222 14712 tablet_service.cc:1514] Processing DeleteTablet for tablet df0fe28e66344ebdb74d7a08e3d99347 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:01:13 UTC) from {username='kudu', principal='kudu/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:54846
I20260926 01:01:13.677829 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 14646
I20260926 01:01:13.680298 14851 tablet_service.cc:1514] Processing DeleteTablet for tablet df0fe28e66344ebdb74d7a08e3d99347 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:01:13 UTC) from {username='kudu', principal='kudu/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:60092
I20260926 01:01:13.683694 15096 tablet_replica.cc:330] T df0fe28e66344ebdb74d7a08e3d99347 P 8a2911e3464044b495e22fedceecf94a: stopping tablet replica
I20260926 01:01:13.683842 15096 raft_consensus.cc:2229] T df0fe28e66344ebdb74d7a08e3d99347 P 8a2911e3464044b495e22fedceecf94a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:01:13.684006 15096 raft_consensus.cc:2258] T df0fe28e66344ebdb74d7a08e3d99347 P 8a2911e3464044b495e22fedceecf94a [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 01:01:13.685125 14549 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.75.1:43573 (error 108)
I20260926 01:01:13.685379 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 14785
I20260926 01:01:13.689738 15065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:01:13.689608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.1:43573 (local address 127.0.0.1:54864)
0926 01:01:13.689657 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 01:01:13.689709 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.1:43573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260926 01:01:13.689888 14549 catalog_manager.cc:4584] TS 7695a4a9cf324c1f930cbd5b5cb397be (127.3.75.1:43573): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet df0fe28e66344ebdb74d7a08e3d99347: Network error: Client connection negotiation failed: client connection to 127.3.75.1:43573: connect: Connection refused (error 111)
I20260926 01:01:13.690277 15096 ts_tablet_manager.cc:1905] T df0fe28e66344ebdb74d7a08e3d99347 P 8a2911e3464044b495e22fedceecf94a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:01:13.692049 15096 ts_tablet_manager.cc:1918] T df0fe28e66344ebdb74d7a08e3d99347 P 8a2911e3464044b495e22fedceecf94a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:01:13.692173 15096 log.cc:1196] T df0fe28e66344ebdb74d7a08e3d99347 P 8a2911e3464044b495e22fedceecf94a: Deleting WAL directory at /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384372073861-3372-0/minicluster-data/ts-2/wal/wals/df0fe28e66344ebdb74d7a08e3d99347
I20260926 01:01:13.692807 15096 ts_tablet_manager.cc:1939] T df0fe28e66344ebdb74d7a08e3d99347 P 8a2911e3464044b495e22fedceecf94a: Deleting consensus metadata
I20260926 01:01:13.693801 14550 catalog_manager.cc:4841] TS 8a2911e3464044b495e22fedceecf94a (127.3.75.3:44709): tablet df0fe28e66344ebdb74d7a08e3d99347 (table test-table [id=ab997f2d454049a6bc4c75f8c2d95e74]) successfully deleted
I20260926 01:01:13.694931 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 14923
I20260926 01:01:13.694977 15065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:01:13.694850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.2:39331 (local address 127.0.0.1:60100)
0926 01:01:13.694914 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0926 01:01:13.694945 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.2:39331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
Sep 26 01:01:13 dist-test-slave-7v10 krb5kdc[14089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384472, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.62@KRBTEST.COM for HTTP/127.3.75.20@KRBTEST.COM
I20260926 01:01:13.704866 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 14533
I20260926 01:01:13.711616 3372 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 01:01:14.056131 3372 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 01:01:14.056233 3372 mini_ranger.cc:66] Stopping Ranger...
I20260926 01:01:14.401304 3372 mini_ranger.cc:68] Stopped Ranger
2026-09-26 01:01:14.401 UTC [14119] LOG: received smart shutdown request
2026-09-26 01:01:14.404 UTC [14119] LOG: background worker "logical replication launcher" (PID 14128) exited with exit code 1
2026-09-26 01:01:14.405 UTC [14123] LOG: shutting down
2026-09-26 01:01:14.566 UTC [14119] LOG: database system is shut down
2026-09-26T01:01:14Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (44891 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:01:14 dist-test-slave-7v10 krb5kdc[15109](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:01:14 dist-test-slave-7v10 krb5kdc[15109](info): set up 2 sockets
Sep 26 01:01:14 dist-test-slave-7v10 krb5kdc[15109](info): commencing operation
krb5kdc: starting...
W20260926 01:01:15.745880 3372 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Sep 26 01:01:15 dist-test-slave-7v10 krb5kdc[15109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:01:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:01:15Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.75.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.75.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.75.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/krb5kdc/rangeradmin_127.3.75.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.75.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/krb5kdc/rangeradmin_127.3.75.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.75.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.75.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.75.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/krb5kdc/rangerlookup_127.3.75.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.75.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/krb5kdc/rangerlookup_127.3.75.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.75.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.75.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.75.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/krb5kdc/HTTP_127.3.75.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.75.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/krb5kdc/HTTP_127.3.75.20@KRBTEST.COM.keytab.
I20260926 01:01:15.899466 3372 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasktuUsJs/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/postgres -l logfile start
2026-09-26 01:01:18.325 UTC [15139] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 01:01:18.325 UTC [15139] LOG: listening on IPv4 address "127.3.75.20", port 49405
2026-09-26 01:01:18.328 UTC [15139] LOG: listening on Unix socket "/tmp/.s.PGSQL.49405"
2026-09-26 01:01:18.363 UTC [15142] LOG: database system was shut down at 2026-09-26 01:01:16 UTC
2026-09-26 01:01:18.368 UTC [15139] LOG: database system is ready to accept connections
I20260926 01:01:19.361822 3372 mini_postgres.cc:96] Postgres bound to 49405
2026-09-26 01:01:19.367 UTC [15150] FATAL: database "slave" does not exist
127.3.75.20:49405 - accepting connections
I20260926 01:01:19.368520 3372 mini_ranger.cc:161] Starting Ranger...
I20260926 01:01:19.382843 3372 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 01:01:19.858647 3372 mini_ranger.cc:87] Created ranger Postgres database
I20260926 01:01:19.858741 3372 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ranger-admin
2026-09-26 01:01:20,124 [I] DB FLAVOR :POSTGRES
2026-09-26 01:01:20,124 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 01:01:20,124 [I] Checking connection..
2026-09-26 01:01:20,125 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktuUsJs/build/release/bin/postgresql.jar:/tmp/dist-test-tasktuUsJs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.75.20:49405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 01:01:20,368 [I] Checking connection passed.
2026-09-26 01:01:20,369 [I] --------- Verifying version history table ---------
2026-09-26 01:01:20,369 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktuUsJs/build/release/bin/postgresql.jar:/tmp/dist-test-tasktuUsJs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.75.20:49405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 01:01:20,597 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 01:01:20,597 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktuUsJs/build/release/bin/postgresql.jar:/tmp/dist-test-tasktuUsJs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.75.20:49405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 01:01:20,877 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 01:01:20,877 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 01:01:20,878 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktuUsJs/build/release/bin/postgresql.jar:/tmp/dist-test-tasktuUsJs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.75.20:49405/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasktuUsJs/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 01:01:21.105 UTC [15214] WARNING: there is no transaction in progress
2026-09-26 01:01:21,112 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 01:01:21,112 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktuUsJs/build/release/bin/postgresql.jar:/tmp/dist-test-tasktuUsJs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.75.20:49405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 01:01:21,361 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 01:01:21,362 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 01:01:21,362 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktuUsJs/build/release/bin/postgresql.jar:/tmp/dist-test-tasktuUsJs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.75.20:49405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 01:01:21,602 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktuUsJs/build/release/bin/postgresql.jar:/tmp/dist-test-tasktuUsJs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.75.20:49405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 01:01:21,869 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktuUsJs/build/release/bin/postgresql.jar:/tmp/dist-test-tasktuUsJs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.75.20:49405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-7v10.c.gcp-upstream.internal','N') ;"
2026-09-26 01:01:22,119 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktuUsJs/build/release/bin/postgresql.jar:/tmp/dist-test-tasktuUsJs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.75.20:49405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 01:01:22,345 [I] Table x_portal_user does not exist in database ranger
2026-09-26 01:01:22,345 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktuUsJs/build/release/bin/postgresql.jar:/tmp/dist-test-tasktuUsJs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.75.20:49405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 01:01:22,567 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 01:01:22,568 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktuUsJs/build/release/bin/postgresql.jar:/tmp/dist-test-tasktuUsJs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.75.20:49405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 01:01:22,816 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktuUsJs/build/release/bin/postgresql.jar:/tmp/dist-test-tasktuUsJs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.75.20:49405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 01:01:23,043 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 01:01:23,044 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktuUsJs/build/release/bin/postgresql.jar:/tmp/dist-test-tasktuUsJs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.75.20:49405/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasktuUsJs/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 01:01:23.971 UTC [15322] WARNING: there is no transaction in progress
2026-09-26 01:01:23.986 UTC [15322] WARNING: there is no transaction in progress
2026-09-26 01:01:24.001 UTC [15322] WARNING: there is no transaction in progress
2026-09-26 01:01:24.015 UTC [15322] WARNING: there is no transaction in progress
2026-09-26 01:01:24.030 UTC [15322] WARNING: there is no transaction in progress
2026-09-26 01:01:24.045 UTC [15322] WARNING: there is no transaction in progress
2026-09-26 01:01:24.097 UTC [15322] WARNING: there is no transaction in progress
2026-09-26 01:01:24.104 UTC [15322] WARNING: there is no transaction in progress
2026-09-26 01:01:24.119 UTC [15322] WARNING: there is no transaction in progress
2026-09-26 01:01:24.128 UTC [15322] WARNING: there is no transaction in progress
2026-09-26 01:01:24.138 UTC [15322] WARNING: there is no transaction in progress
2026-09-26 01:01:24.152 UTC [15322] WARNING: there is no transaction in progress
2026-09-26 01:01:24.162 UTC [15322] WARNING: there is no transaction in progress
2026-09-26 01:01:24.168 UTC [15322] WARNING: there is no transaction in progress
2026-09-26 01:01:24.174 UTC [15322] WARNING: there is no transaction in progress
2026-09-26 01:01:24.180 UTC [15322] WARNING: there is no transaction in progress
2026-09-26 01:01:24,784 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 01:01:24,784 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktuUsJs/build/release/bin/postgresql.jar:/tmp/dist-test-tasktuUsJs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.75.20:49405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 01:01:25,035 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 01:01:25,035 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktuUsJs/build/release/bin/postgresql.jar:/tmp/dist-test-tasktuUsJs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.75.20:49405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 01:01:25,296 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 01:01:25,296 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktuUsJs/build/release/bin/postgresql.jar:/tmp/dist-test-tasktuUsJs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.75.20:49405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 01:01:25,550 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 01:01:25,550 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktuUsJs/build/release/bin/postgresql.jar:/tmp/dist-test-tasktuUsJs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.75.20:49405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 01:01:25,774 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktuUsJs/build/release/bin/postgresql.jar:/tmp/dist-test-tasktuUsJs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.75.20:49405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 01:01:26,021 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktuUsJs/build/release/bin/postgresql.jar:/tmp/dist-test-tasktuUsJs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.75.20:49405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-7v10.c.gcp-upstream.internal';"
2026-09-26 01:01:26,238 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 01:01:26,238 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktuUsJs/build/release/bin/postgresql.jar:/tmp/dist-test-tasktuUsJs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.75.20:49405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 01:01:26,497 [I] DB_PATCHES have already been applied
I20260926 01:01:26.502600 3372 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ranger-admin:/tmp/dist-test-tasktuUsJs/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasktuUsJs/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasktuUsJs/build/release/bin/postgresql.jar:/tmp/dist-test-tasktuUsJs/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 01:01:26.502681 3372 mini_ranger.cc:193] Using host: 127.3.75.20
I20260926 01:01:26.505856 3372 mini_ranger.cc:227] Ranger admin URL: http://127.3.75.20:56827
Sep 26, 2026 1:01:26 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 1:01:26 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 1:01:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 1:01:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 1:01:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 1:01:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 1:01:27 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 01:01:27.612546 3372 mini_ranger.cc:160] Time spent starting Ranger: real 8.244s user 0.000s sys 0.009s
Sep 26, 2026 1:01:47 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 1:01:47 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 26, 2026 1:01:47 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 1:01:47 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 1:01:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 1:01:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 1:01:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 1:01:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 1:01:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 1:01:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 1:01:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 1:01:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 1:01:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 1:01:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 1:01:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 1:01:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 1:01:47 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 1:01:48 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 01:01:49.141338 3372 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.75.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.75.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.75.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/krb5kdc/rangerkms_127.3.75.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.75.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/krb5kdc/rangerkms_127.3.75.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.75.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.75.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.75.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/krb5kdc/HTTP_127.3.75.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.75.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/krb5kdc/HTTP_127.3.75.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 01:01:49 dist-test-slave-7v10 krb5kdc[15109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384509, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 01:01:49.225930 3372 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 01:01:49.234725 3372 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 01:01:50.094950 3372 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 01:01:50.095044 3372 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ranger-kms
I20260926 01:01:50.095058 3372 mini_ranger_kms.cc:227] Using postgres at 127.3.75.20:49405
2026-09-26 01:01:50,258 [I] DB FLAVOR :POSTGRES
2026-09-26 01:01:50,259 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 01:01:50,259 [I] Checking connection
2026-09-26 01:01:50,259 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktuUsJs/build/release/bin/postgresql.jar:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.75.20:49405/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 01:01:50,503 [I] connection success
2026-09-26 01:01:50,504 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 01:01:50,504 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 01:01:50,504 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktuUsJs/build/release/bin/postgresql.jar:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.75.20:49405/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 01:01:50,731 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 01:01:50,732 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 01:01:50,732 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 01:01:50,732 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktuUsJs/build/release/bin/postgresql.jar:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.75.20:49405/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 01:01:50,996 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 01:01:51.414673 3372 mini_ranger_kms.cc:310] Created kms service
I20260926 01:01:51.625458 3372 mini_ranger_kms.cc:326] Created kudu user
I20260926 01:01:51.669386 3372 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 01:01:51 dist-test-slave-7v10 krb5kdc[15109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384509, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.75.20@KRBTEST.COM
I20260926 01:01:52.057121 3372 mini_ranger_kms.cc:384] Added ranger policy
I20260926 01:01:52.057229 3372 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ranger-kms:/tmp/dist-test-tasktuUsJs/build/release/bin/postgresql.jar:/tmp/dist-test-tasktuUsJs/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasktuUsJs/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 01:01:52.057248 3372 mini_ranger_kms.cc:240] Using host: 127.3.75.20
I20260926 01:01:52.060592 3372 mini_ranger_kms.cc:276] Ranger KMS PID: 15502
I20260926 01:01:52.060669 3372 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.75.20:36891
Sep 26, 2026 1:01:52 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 1:01:52 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 1:01:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 1:01:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 1:01:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 1:01:53 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 26 01:01:54 dist-test-slave-7v10 krb5kdc[15109](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790384514, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.75.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:01:54.339491 3372 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.114s user 0.006s sys 0.140s
I20260926 01:01:54.339613 3372 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 01:01:54.339663 3372 mini_ranger_kms.cc:401] 127.3.75.20:36891/kms/v1/keys
Sep 26 01:01:54 dist-test-slave-7v10 krb5kdc[15109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384514, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.75.20@KRBTEST.COM for HTTP/127.3.75.20@KRBTEST.COM
Sep 26 01:01:54 dist-test-slave-7v10 krb5kdc[15109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384514, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.75.20@KRBTEST.COM for HTTP/127.3.75.20@KRBTEST.COM
Sep 26 01:01:57 dist-test-slave-7v10 krb5kdc[15109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.75.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab.
Entry for principal kudu/127.3.75.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab.
I20260926 01:01:57.071194 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:42685
--webserver_interface=127.3.75.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:33405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.75.62:42685
--ranger_config_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.75.20:36891/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:01:57.167984 15553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:01:57.168179 15553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:01:57.168200 15553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:01:57.169273 15553 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:01:57.169312 15553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:01:57.169324 15553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:01:57.169335 15553 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:01:57.169346 15553 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:01:57.170804 15553 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:33405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.75.20:36891/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.75.62:42685
--ranger_config_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:42685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:01:57.171052 15553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:01:57.171275 15553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:01:57.176208 15558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:01:57.176230 15559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:01:57.176316 15553 server_base.cc:1034] running on GCE node
W20260926 01:01:57.176208 15561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:01:57.176657 15553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:01:57.176887 15553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:01:57.178028 15553 hybrid_clock.cc:648] HybridClock initialized: now 1790384517178006 us; error 37 us; skew 500 ppm
Sep 26 01:01:57 dist-test-slave-7v10 krb5kdc[15109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384517, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:01:57.180544 15553 init.cc:377] Logged in from keytab as kudu/127.3.75.62@KRBTEST.COM (short username kudu)
I20260926 01:01:57.180938 15553 webserver.cc:460] Webserver started at http://127.3.75.62:42109/ using document root <none> and password file <none>
I20260926 01:01:57.181154 15553 fs_manager.cc:359] Metadata directory not provided
I20260926 01:01:57.181192 15553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:01:57.181352 15553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 01:01:57 dist-test-slave-7v10 krb5kdc[15109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384517, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.62@KRBTEST.COM for HTTP/127.3.75.20@KRBTEST.COM
I20260926 01:01:57.241719 15553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/master-0/data/instance:
uuid: "e025374659ce4c9c843075e6ce79ee28"
format_stamp: "Formatted at 2026-09-26 01:01:57 on dist-test-slave-7v10"
server_key: "882b1f7f2c1ef8b2e461d368b1437eb5"
server_key_iv: "8d81a27e8456a93cad09a859adca8eb2"
server_key_version: "kuduclusterkey@0"
I20260926 01:01:57.242166 15553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/master-0/wal/instance:
uuid: "e025374659ce4c9c843075e6ce79ee28"
format_stamp: "Formatted at 2026-09-26 01:01:57 on dist-test-slave-7v10"
server_key: "882b1f7f2c1ef8b2e461d368b1437eb5"
server_key_iv: "8d81a27e8456a93cad09a859adca8eb2"
server_key_version: "kuduclusterkey@0"
I20260926 01:01:57.243924 15553 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260926 01:01:57.258308 15568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:01:57.263149 15553 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260926 01:01:57.263283 15553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/master-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/master-0/wal
uuid: "e025374659ce4c9c843075e6ce79ee28"
format_stamp: "Formatted at 2026-09-26 01:01:57 on dist-test-slave-7v10"
server_key: "882b1f7f2c1ef8b2e461d368b1437eb5"
server_key_iv: "8d81a27e8456a93cad09a859adca8eb2"
server_key_version: "kuduclusterkey@0"
I20260926 01:01:57.263399 15553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:01:57.303676 15553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:01:57.305775 15553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:01:57.305949 15553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:01:57.310806 15553 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.62:42685
I20260926 01:01:57.310895 15621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.62:42685 every 8 connection(s)
I20260926 01:01:57.311269 15553 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
I20260926 01:01:57.311988 15622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:01:57.314271 15622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e025374659ce4c9c843075e6ce79ee28: Bootstrap starting.
I20260926 01:01:57.314860 15622 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e025374659ce4c9c843075e6ce79ee28: Neither blocks nor log segments found. Creating new log.
I20260926 01:01:57.315223 15622 log.cc:824] T 00000000000000000000000000000000 P e025374659ce4c9c843075e6ce79ee28: Log is configured to *not* fsync() on all Append() calls
I20260926 01:01:57.315938 15622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e025374659ce4c9c843075e6ce79ee28: No bootstrap required, opened a new log
I20260926 01:01:57.317862 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 15553
I20260926 01:01:57.317955 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/master-0/wal/instance
I20260926 01:01:57.317958 15622 raft_consensus.cc:348] T 00000000000000000000000000000000 P e025374659ce4c9c843075e6ce79ee28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e025374659ce4c9c843075e6ce79ee28" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42685 } }
I20260926 01:01:57.318140 3372 external_mini_cluster.cc:1428] Setting key a20135550634d298ce4bf9429b69549f
I20260926 01:01:57.318156 15622 raft_consensus.cc:374] T 00000000000000000000000000000000 P e025374659ce4c9c843075e6ce79ee28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:01:57.318222 15622 raft_consensus.cc:729] T 00000000000000000000000000000000 P e025374659ce4c9c843075e6ce79ee28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e025374659ce4c9c843075e6ce79ee28, State: Initialized, Role: FOLLOWER
I20260926 01:01:57.318389 15622 consensus_queue.cc:260] T 00000000000000000000000000000000 P e025374659ce4c9c843075e6ce79ee28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e025374659ce4c9c843075e6ce79ee28" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42685 } }
I20260926 01:01:57.318483 15622 raft_consensus.cc:388] T 00000000000000000000000000000000 P e025374659ce4c9c843075e6ce79ee28 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:01:57.318516 15622 raft_consensus.cc:482] T 00000000000000000000000000000000 P e025374659ce4c9c843075e6ce79ee28 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:01:57.318560 15622 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e025374659ce4c9c843075e6ce79ee28 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:01:57.319211 15622 raft_consensus.cc:504] T 00000000000000000000000000000000 P e025374659ce4c9c843075e6ce79ee28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e025374659ce4c9c843075e6ce79ee28" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42685 } }
I20260926 01:01:57.319415 15622 leader_election.cc:302] T 00000000000000000000000000000000 P e025374659ce4c9c843075e6ce79ee28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e025374659ce4c9c843075e6ce79ee28; no voters:
I20260926 01:01:57.319595 15622 leader_election.cc:288] T 00000000000000000000000000000000 P e025374659ce4c9c843075e6ce79ee28 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:01:57.319638 15627 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e025374659ce4c9c843075e6ce79ee28 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:01:57.319903 15627 raft_consensus.cc:686] T 00000000000000000000000000000000 P e025374659ce4c9c843075e6ce79ee28 [term 1 LEADER]: Becoming Leader. State: Replica: e025374659ce4c9c843075e6ce79ee28, State: Running, Role: LEADER
I20260926 01:01:57.319963 15622 sys_catalog.cc:564] T 00000000000000000000000000000000 P e025374659ce4c9c843075e6ce79ee28 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:01:57.320039 15627 consensus_queue.cc:237] T 00000000000000000000000000000000 P e025374659ce4c9c843075e6ce79ee28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e025374659ce4c9c843075e6ce79ee28" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42685 } }
I20260926 01:01:57.320418 15627 sys_catalog.cc:455] T 00000000000000000000000000000000 P e025374659ce4c9c843075e6ce79ee28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e025374659ce4c9c843075e6ce79ee28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e025374659ce4c9c843075e6ce79ee28" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42685 } } }
I20260926 01:01:57.320542 15627 sys_catalog.cc:458] T 00000000000000000000000000000000 P e025374659ce4c9c843075e6ce79ee28 [sys.catalog]: This master's current role is: LEADER
I20260926 01:01:57.320705 15627 sys_catalog.cc:455] T 00000000000000000000000000000000 P e025374659ce4c9c843075e6ce79ee28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e025374659ce4c9c843075e6ce79ee28. Latest consensus state: current_term: 1 leader_uuid: "e025374659ce4c9c843075e6ce79ee28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e025374659ce4c9c843075e6ce79ee28" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42685 } } }
I20260926 01:01:57.320802 15627 sys_catalog.cc:458] T 00000000000000000000000000000000 P e025374659ce4c9c843075e6ce79ee28 [sys.catalog]: This master's current role is: LEADER
I20260926 01:01:57.320945 15635 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:01:57.321424 15635 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:01:57.321710 15622 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Sep 26 01:01:57 dist-test-slave-7v10 krb5kdc[15109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 01:01:57.331378 15635 catalog_manager.cc:1347] Generated new cluster ID: 50d4d503066b41acbc2cf679a5247952
I20260926 01:01:57.331454 15635 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:01:57.332356 15626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:01:57.318587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40828 (local address 127.3.75.62:42685)
0926 01:01:57.318909 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0926 01:01:57.318918 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 01:01:57.318952 (+ 34us) server_negotiation.cc:407] Connection header received
0926 01:01:57.319067 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:01:57.319074 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:01:57.319129 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:01:57.319274 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:01:57.320119 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:01:57.321664 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:01:57.323667 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:01:57.324991 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:01:57.325174 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:01:57.328192 (+ 3018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:01:57.328196 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:01:57.328196 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:01:57.328207 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:01:57.330462 (+ 2255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:01:57.330848 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:01:57.330851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:01:57.330851 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:01:57.330988 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:01:57.331195 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:01:57.331196 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:01:57.331196 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:01:57.331490 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:01:57.331614 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 01:01:57.331880 (+ 266us) server_negotiation.cc:299] Negotiation successful
0926 01:01:57.332065 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":253,"thread_start_us":80,"threads_started":1}
I20260926 01:01:57.357411 15635 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:01:57.358207 15635 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:01:57.370008 15635 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e025374659ce4c9c843075e6ce79ee28: Generated new TSK 0
I20260926 01:01:57.370203 15635 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.75.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab.
Entry for principal kudu/127.3.75.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab.
I20260926 01:01:58.093938 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.75.20:36891/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.1:0
--local_ip_for_outbound_sockets=127.3.75.1
--webserver_interface=127.3.75.1
--webserver_port=0
--tserver_master_addrs=127.3.75.62:42685
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab
--principal=kudu/127.3.75.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:33405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:01:58.207175 15667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:01:58.207469 15667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:01:58.207506 15667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:01:58.209218 15667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:01:58.209276 15667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:01:58.209316 15667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.1
I20260926 01:01:58.211333 15667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:33405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.75.20:36891/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab
--principal=kudu/127.3.75.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.75.1
--webserver_port=0
--tserver_master_addrs=127.3.75.62:42685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.1
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:01:58.211583 15667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:01:58.211833 15667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:01:58.217941 15672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:01:58.217902 15673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:01:58.218132 15667 server_base.cc:1034] running on GCE node
W20260926 01:01:58.218220 15675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:01:58.218436 15667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:01:58.218686 15667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:01:58.219861 15667 hybrid_clock.cc:648] HybridClock initialized: now 1790384518219819 us; error 54 us; skew 500 ppm
Sep 26 01:01:58 dist-test-slave-7v10 krb5kdc[15109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384518, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:01:58.222750 15667 init.cc:377] Logged in from keytab as kudu/127.3.75.1@KRBTEST.COM (short username kudu)
I20260926 01:01:58.223317 15667 webserver.cc:460] Webserver started at http://127.3.75.1:40983/ using document root <none> and password file <none>
I20260926 01:01:58.223583 15667 fs_manager.cc:359] Metadata directory not provided
I20260926 01:01:58.223652 15667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:01:58.223793 15667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 01:01:58 dist-test-slave-7v10 krb5kdc[15109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384518, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.1@KRBTEST.COM for HTTP/127.3.75.20@KRBTEST.COM
I20260926 01:01:58.237658 15667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-0/data/instance:
uuid: "bf72ecfdc187462ca6275a2655fd74b4"
format_stamp: "Formatted at 2026-09-26 01:01:58 on dist-test-slave-7v10"
server_key: "5bfec010f648e5b1acbbb39e324352fc"
server_key_iv: "3e31ca364548e4ee2f7ffdd415bdfa01"
server_key_version: "kuduclusterkey@0"
I20260926 01:01:58.238435 15667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-0/wal/instance:
uuid: "bf72ecfdc187462ca6275a2655fd74b4"
format_stamp: "Formatted at 2026-09-26 01:01:58 on dist-test-slave-7v10"
server_key: "5bfec010f648e5b1acbbb39e324352fc"
server_key_iv: "3e31ca364548e4ee2f7ffdd415bdfa01"
server_key_version: "kuduclusterkey@0"
I20260926 01:01:58.241423 15667 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260926 01:01:58.256583 15683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:01:58.256845 15667 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 01:01:58.256999 15667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-0/wal
uuid: "bf72ecfdc187462ca6275a2655fd74b4"
format_stamp: "Formatted at 2026-09-26 01:01:58 on dist-test-slave-7v10"
server_key: "5bfec010f648e5b1acbbb39e324352fc"
server_key_iv: "3e31ca364548e4ee2f7ffdd415bdfa01"
server_key_version: "kuduclusterkey@0"
I20260926 01:01:58.257158 15667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:01:58.277843 15667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:01:58.280412 15667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:01:58.280596 15667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:01:58.280967 15667 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:01:58.281383 15667 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:01:58.281436 15667 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:01:58.281469 15667 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:01:58.281526 15667 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:01:58.288915 15667 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.1:37121
I20260926 01:01:58.289003 15799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.1:37121 every 8 connection(s)
I20260926 01:01:58.289350 15667 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
I20260926 01:01:58.289916 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 15667
I20260926 01:01:58.290005 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-0/wal/instance
I20260926 01:01:58.290244 3372 external_mini_cluster.cc:1428] Setting key 71d4ea3adc62cf9b869199b4186978d6
Sep 26 01:01:58 dist-test-slave-7v10 krb5kdc[15109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384518, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.1@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.75.2@KRBTEST.COM; defaulting to no policy
I20260926 01:01:58.301630 15802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:01:58.290058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:42685 (local address 127.3.75.1:49929)
0926 01:01:58.290257 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 01:01:58.290359 (+ 102us) client_negotiation.cc:174] Beginning negotiation
0926 01:01:58.290954 (+ 595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:01:58.292605 (+ 1651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:01:58.292611 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:01:58.292880 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:01:58.293177 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:01:58.293183 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:01:58.294023 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:01:58.294024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:01:58.294927 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:01:58.294929 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:01:58.295377 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:01:58.295379 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:01:58.295482 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:01:58.296116 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:01:58.296126 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:01:58.297817 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:01:58.300001 (+ 2184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:01:58.300007 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:01:58.300009 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:01:58.300307 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:01:58.300524 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:01:58.300525 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:01:58.300525 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:01:58.300649 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:01:58.300953 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:01:58.300959 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:01:58.301131 (+ 172us) client_negotiation.cc:769] Sending connection context
0926 01:01:58.301240 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 01:01:58.301391 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":107,"threads_started":1}
I20260926 01:01:58.302117 15803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:01:58.290466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:49929 (local address 127.3.75.62:42685)
0926 01:01:58.290631 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 01:01:58.290633 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:01:58.290644 (+ 11us) server_negotiation.cc:407] Connection header received
0926 01:01:58.291047 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:01:58.291048 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:01:58.291060 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:01:58.291125 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:01:58.293316 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:01:58.293926 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:01:58.295030 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:01:58.295232 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:01:58.295301 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:01:58.297942 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:01:58.297943 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:01:58.297944 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:01:58.297951 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:01:58.299905 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:01:58.300396 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:01:58.300398 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:01:58.300398 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:01:58.300459 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:01:58.300716 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:01:58.300717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:01:58.300718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:01:58.300868 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:01:58.300926 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 01:01:58.301935 (+ 1009us) server_negotiation.cc:299] Negotiation successful
0926 01:01:58.302022 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":86,"threads_started":1}
I20260926 01:01:58.302517 15800 heartbeater.cc:347] Connected to a master server at 127.3.75.62:42685
I20260926 01:01:58.302615 15800 heartbeater.cc:464] Registering TS with master...
I20260926 01:01:58.302800 15800 heartbeater.cc:511] Master 127.3.75.62:42685 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.2@KRBTEST.COM" created.
I20260926 01:01:58.303408 15585 ts_manager.cc:194] Registered new tserver with Master: bf72ecfdc187462ca6275a2655fd74b4 (127.3.75.1:37121)
I20260926 01:01:58.304242 15585 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.75.1@KRBTEST.COM'} at 127.3.75.1:49929
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab.
Entry for principal kudu/127.3.75.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab.
I20260926 01:01:58.321834 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.75.20:36891/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.2:0
--local_ip_for_outbound_sockets=127.3.75.2
--webserver_interface=127.3.75.2
--webserver_port=0
--tserver_master_addrs=127.3.75.62:42685
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab
--principal=kudu/127.3.75.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:33405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 01:01:58 dist-test-slave-7v10 krb5kdc[15109](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790384518, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 01:01:58.436932 15806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:01:58.437178 15806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:01:58.437209 15806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:01:58.438905 15806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:01:58.438956 15806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:01:58.438994 15806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.2
I20260926 01:01:58.441238 15806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:33405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.75.20:36891/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab
--principal=kudu/127.3.75.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.75.2
--webserver_port=0
--tserver_master_addrs=127.3.75.62:42685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.2
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:01:58.441476 15806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:01:58.441781 15806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:01:58.448129 15816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:01:58.448129 15814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:01:58.448354 15813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:01:58.449395 15806 server_base.cc:1034] running on GCE node
I20260926 01:01:58.449592 15806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:01:58.449842 15806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:01:58.450949 15806 hybrid_clock.cc:648] HybridClock initialized: now 1790384518450928 us; error 35 us; skew 500 ppm
Sep 26 01:01:58 dist-test-slave-7v10 krb5kdc[15109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384518, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:01:58.454120 15806 init.cc:377] Logged in from keytab as kudu/127.3.75.2@KRBTEST.COM (short username kudu)
I20260926 01:01:58.454592 15806 webserver.cc:460] Webserver started at http://127.3.75.2:32899/ using document root <none> and password file <none>
I20260926 01:01:58.454865 15806 fs_manager.cc:359] Metadata directory not provided
I20260926 01:01:58.454941 15806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:01:58.455055 15806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 01:01:58 dist-test-slave-7v10 krb5kdc[15109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384518, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.2@KRBTEST.COM for HTTP/127.3.75.20@KRBTEST.COM
I20260926 01:01:58.472783 15806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-1/data/instance:
uuid: "4bd73316892d4aa7b951b06f3551e963"
format_stamp: "Formatted at 2026-09-26 01:01:58 on dist-test-slave-7v10"
server_key: "f110abdec298efd821657cf85731f25d"
server_key_iv: "00e8def8d9098142dae0ab69dbd0d6a8"
server_key_version: "kuduclusterkey@0"
I20260926 01:01:58.473172 15806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-1/wal/instance:
uuid: "4bd73316892d4aa7b951b06f3551e963"
format_stamp: "Formatted at 2026-09-26 01:01:58 on dist-test-slave-7v10"
server_key: "f110abdec298efd821657cf85731f25d"
server_key_iv: "00e8def8d9098142dae0ab69dbd0d6a8"
server_key_version: "kuduclusterkey@0"
I20260926 01:01:58.480149 15806 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.000s
I20260926 01:01:58.499008 15823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:01:58.499187 15806 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.001s sys 0.000s
I20260926 01:01:58.499269 15806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-1/wal
uuid: "4bd73316892d4aa7b951b06f3551e963"
format_stamp: "Formatted at 2026-09-26 01:01:58 on dist-test-slave-7v10"
server_key: "f110abdec298efd821657cf85731f25d"
server_key_iv: "00e8def8d9098142dae0ab69dbd0d6a8"
server_key_version: "kuduclusterkey@0"
I20260926 01:01:58.499372 15806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:01:58.515276 15806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:01:58.517292 15806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:01:58.517457 15806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:01:58.517705 15806 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:01:58.518075 15806 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:01:58.518131 15806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:01:58.518170 15806 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:01:58.518195 15806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:01:58.525877 15806 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.2:46241
I20260926 01:01:58.526216 15806 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
I20260926 01:01:58.526865 15937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.2:46241 every 8 connection(s)
I20260926 01:01:58.528776 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 15806
I20260926 01:01:58.528889 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-1/wal/instance
I20260926 01:01:58.529063 3372 external_mini_cluster.cc:1428] Setting key db3a81f4e8b2c5f20b4f56d27d1bd877
Sep 26 01:01:58 dist-test-slave-7v10 krb5kdc[15109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384518, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.2@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 01:01:58.539505 15803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:01:58.527047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:44625 (local address 127.3.75.62:42685)
0926 01:01:58.527096 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 01:01:58.527098 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:01:58.529253 (+ 2155us) server_negotiation.cc:407] Connection header received
0926 01:01:58.529293 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:01:58.529294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:01:58.529307 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:01:58.529359 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:01:58.530169 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:01:58.530962 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:01:58.532243 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:01:58.532560 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:01:58.532607 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:01:58.535206 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:01:58.535207 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:01:58.535208 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:01:58.535217 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:01:58.537792 (+ 2575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:01:58.538280 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:01:58.538281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:01:58.538282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:01:58.538355 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:01:58.538699 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:01:58.538700 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:01:58.538701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:01:58.538851 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:01:58.538967 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 01:01:58.539304 (+ 337us) server_negotiation.cc:299] Negotiation successful
0926 01:01:58.539401 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 01:01:58.539561 15940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:01:58.527186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:42685 (local address 127.3.75.2:44625)
0926 01:01:58.527341 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 01:01:58.527383 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 01:01:58.528053 (+ 670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:01:58.529491 (+ 1438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:01:58.529495 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:01:58.529758 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:01:58.530065 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:01:58.530072 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:01:58.531076 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:01:58.531077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:01:58.532140 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:01:58.532143 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:01:58.532711 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:01:58.532712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:01:58.532863 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:01:58.533460 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:01:58.533470 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:01:58.535102 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:01:58.537886 (+ 2784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:01:58.537892 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:01:58.537894 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:01:58.538197 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:01:58.538429 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:01:58.538430 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:01:58.538430 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:01:58.538583 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:01:58.538958 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:01:58.538964 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:01:58.539148 (+ 184us) client_negotiation.cc:769] Sending connection context
0926 01:01:58.539255 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 01:01:58.539373 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":88,"threads_started":1}
I20260926 01:01:58.540030 15938 heartbeater.cc:347] Connected to a master server at 127.3.75.62:42685
I20260926 01:01:58.540124 15938 heartbeater.cc:464] Registering TS with master...
I20260926 01:01:58.540326 15938 heartbeater.cc:511] Master 127.3.75.62:42685 requested a full tablet report, sending...
I20260926 01:01:58.541344 15585 ts_manager.cc:194] Registered new tserver with Master: 4bd73316892d4aa7b951b06f3551e963 (127.3.75.2:46241)
I20260926 01:01:58.541831 15585 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.75.2@KRBTEST.COM'} at 127.3.75.2:44625
WARNING: no policy specified for kudu/127.3.75.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab.
Entry for principal kudu/127.3.75.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab.
I20260926 01:01:58.561798 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.75.20:36891/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.3:0
--local_ip_for_outbound_sockets=127.3.75.3
--webserver_interface=127.3.75.3
--webserver_port=0
--tserver_master_addrs=127.3.75.62:42685
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab
--principal=kudu/127.3.75.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:33405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:01:58.683017 15943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:01:58.683303 15943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:01:58.683347 15943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:01:58.685029 15943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:01:58.685084 15943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:01:58.685127 15943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.3
I20260926 01:01:58.687280 15943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:33405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.3.75.20:36891/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab
--principal=kudu/127.3.75.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.75.3
--webserver_port=0
--tserver_master_addrs=127.3.75.62:42685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.3
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:01:58.687533 15943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:01:58.687799 15943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:01:58.694193 15951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:01:58.694964 15949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:01:58.695160 15948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:01:58.695600 15943 server_base.cc:1034] running on GCE node
I20260926 01:01:58.695815 15943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:01:58.696087 15943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:01:58.697257 15943 hybrid_clock.cc:648] HybridClock initialized: now 1790384518697238 us; error 40 us; skew 500 ppm
Sep 26 01:01:58 dist-test-slave-7v10 krb5kdc[15109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384518, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:01:58.699854 15943 init.cc:377] Logged in from keytab as kudu/127.3.75.3@KRBTEST.COM (short username kudu)
I20260926 01:01:58.700320 15943 webserver.cc:460] Webserver started at http://127.3.75.3:45367/ using document root <none> and password file <none>
I20260926 01:01:58.700562 15943 fs_manager.cc:359] Metadata directory not provided
I20260926 01:01:58.700641 15943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:01:58.700785 15943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 01:01:58 dist-test-slave-7v10 krb5kdc[15109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384518, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.3@KRBTEST.COM for HTTP/127.3.75.20@KRBTEST.COM
I20260926 01:01:58.713302 15943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-2/data/instance:
uuid: "25cbfa91103e44df90817bd7ad62cb45"
format_stamp: "Formatted at 2026-09-26 01:01:58 on dist-test-slave-7v10"
server_key: "5b3d6c8ca5ee5b190ca377f28fafc8d5"
server_key_iv: "c9621477c9c7d6b0b49ecbf034cf58b3"
server_key_version: "kuduclusterkey@0"
I20260926 01:01:58.713784 15943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-2/wal/instance:
uuid: "25cbfa91103e44df90817bd7ad62cb45"
format_stamp: "Formatted at 2026-09-26 01:01:58 on dist-test-slave-7v10"
server_key: "5b3d6c8ca5ee5b190ca377f28fafc8d5"
server_key_iv: "c9621477c9c7d6b0b49ecbf034cf58b3"
server_key_version: "kuduclusterkey@0"
I20260926 01:01:58.715751 15943 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260926 01:01:58.739315 15958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:01:58.739884 15943 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:01:58.740011 15943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-2/wal
uuid: "25cbfa91103e44df90817bd7ad62cb45"
format_stamp: "Formatted at 2026-09-26 01:01:58 on dist-test-slave-7v10"
server_key: "5b3d6c8ca5ee5b190ca377f28fafc8d5"
server_key_iv: "c9621477c9c7d6b0b49ecbf034cf58b3"
server_key_version: "kuduclusterkey@0"
I20260926 01:01:58.740120 15943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:01:58.756695 15943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:01:58.759305 15943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:01:58.759462 15943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:01:58.759748 15943 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:01:58.760115 15943 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:01:58.760157 15943 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:01:58.760192 15943 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:01:58.760214 15943 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:01:58.769979 15943 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.3:35935
I20260926 01:01:58.770350 16071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.3:35935 every 8 connection(s)
I20260926 01:01:58.770460 15943 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
Sep 26 01:01:58 dist-test-slave-7v10 krb5kdc[15109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384518, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.3@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 01:01:58.779004 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 15943
I20260926 01:01:58.779109 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-2/wal/instance
I20260926 01:01:58.779326 3372 external_mini_cluster.cc:1428] Setting key 711746a68fc4713326895dd8a585e2ff
I20260926 01:01:58.782259 15803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:01:58.771523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.3:53313 (local address 127.3.75.62:42685)
0926 01:01:58.771588 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 01:01:58.771591 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:01:58.771777 (+ 186us) server_negotiation.cc:407] Connection header received
0926 01:01:58.772511 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:01:58.772513 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:01:58.772528 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:01:58.772624 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:01:58.773378 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:01:58.774154 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:01:58.775673 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:01:58.775984 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:01:58.776034 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:01:58.778743 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:01:58.778745 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:01:58.778745 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:01:58.778753 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:01:58.780538 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:01:58.781058 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:01:58.781060 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:01:58.781060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:01:58.781124 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:01:58.781445 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:01:58.781446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:01:58.781446 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:01:58.781585 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:01:58.781661 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 01:01:58.782053 (+ 392us) server_negotiation.cc:299] Negotiation successful
0926 01:01:58.782153 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 01:01:58.782460 16074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:01:58.771251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:42685 (local address 127.3.75.3:53313)
0926 01:01:58.771648 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0926 01:01:58.771692 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 01:01:58.772370 (+ 678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:01:58.772698 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:01:58.772703 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:01:58.772969 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:01:58.773255 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:01:58.773263 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:01:58.774269 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:01:58.774270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:01:58.775561 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:01:58.775565 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:01:58.776144 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:01:58.776145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:01:58.776284 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:01:58.776888 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:01:58.776899 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:01:58.778620 (+ 1721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:01:58.780632 (+ 2012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:01:58.780638 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:01:58.780640 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:01:58.780956 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:01:58.781200 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:01:58.781201 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:01:58.781202 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:01:58.781361 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:01:58.781693 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:01:58.781700 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:01:58.781873 (+ 173us) client_negotiation.cc:769] Sending connection context
0926 01:01:58.782011 (+ 138us) client_negotiation.cc:240] Negotiation successful
0926 01:01:58.782170 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":78,"threads_started":1}
I20260926 01:01:58.782929 16072 heartbeater.cc:347] Connected to a master server at 127.3.75.62:42685
I20260926 01:01:58.783031 16072 heartbeater.cc:464] Registering TS with master...
I20260926 01:01:58.783218 16072 heartbeater.cc:511] Master 127.3.75.62:42685 requested a full tablet report, sending...
I20260926 01:01:58.783612 15585 ts_manager.cc:194] Registered new tserver with Master: 25cbfa91103e44df90817bd7ad62cb45 (127.3.75.3:35935)
I20260926 01:01:58.784101 15585 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.75.3@KRBTEST.COM'} at 127.3.75.3:53313
I20260926 01:01:58.791280 3372 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 01:01:58.792054 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 15667
I20260926 01:01:58.799712 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 15806
I20260926 01:01:58.805554 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 15943
I20260926 01:01:58.810959 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 15553
I20260926 01:01:58.815817 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:42685
--webserver_interface=127.3.75.62
--webserver_port=42109
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:33405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.75.62:42685
--ranger_config_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.75.20:36891/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.75.20:36891/kms with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:01:58.895586 16075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:01:58.895776 16075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:01:58.895798 16075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:01:58.896936 16075 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:01:58.896978 16075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:01:58.896992 16075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:01:58.897024 16075 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:01:58.897051 16075 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:01:58.898557 16075 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:33405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.75.20:36891/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.75.62:42685
--ranger_config_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:42685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.75.62
--webserver_port=42109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:01:58.898783 16075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:01:58.899016 16075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:01:58.903719 16083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:01:58.903730 16081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:01:58.903724 16080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:01:58.903906 16075 server_base.cc:1034] running on GCE node
I20260926 01:01:58.904186 16075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:01:58.904472 16075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:01:58.905643 16075 hybrid_clock.cc:648] HybridClock initialized: now 1790384518905630 us; error 40 us; skew 500 ppm
Sep 26 01:01:58 dist-test-slave-7v10 krb5kdc[15109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384518, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:01:58.908039 16075 init.cc:377] Logged in from keytab as kudu/127.3.75.62@KRBTEST.COM (short username kudu)
I20260926 01:01:58.908515 16075 webserver.cc:460] Webserver started at http://127.3.75.62:42109/ using document root <none> and password file <none>
I20260926 01:01:58.908740 16075 fs_manager.cc:359] Metadata directory not provided
I20260926 01:01:58.908811 16075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 01:01:58 dist-test-slave-7v10 krb5kdc[15109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384518, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.62@KRBTEST.COM for HTTP/127.3.75.20@KRBTEST.COM
I20260926 01:01:58.961201 16075 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:01:58.962005 16091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:01:58.962185 16075 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:01:58.962307 16075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/master-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/master-0/wal
uuid: "e025374659ce4c9c843075e6ce79ee28"
format_stamp: "Formatted at 2026-09-26 01:01:57 on dist-test-slave-7v10"
server_key: "882b1f7f2c1ef8b2e461d368b1437eb5"
server_key_iv: "8d81a27e8456a93cad09a859adca8eb2"
server_key_version: "kuduclusterkey@0"
I20260926 01:01:58.962670 16075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:01:58.975625 16075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:01:58.977277 16075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:01:58.977442 16075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:01:58.982215 16075 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.62:42685
I20260926 01:01:58.982311 16143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.62:42685 every 8 connection(s)
I20260926 01:01:58.982694 16075 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
I20260926 01:01:58.983995 16144 sys_catalog.cc:263] Verifying existing consensus state
I20260926 01:01:58.984620 16144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e025374659ce4c9c843075e6ce79ee28: Bootstrap starting.
I20260926 01:01:58.991405 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 16075
I20260926 01:01:58.991735 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.75.20:36891/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.1:37121
--local_ip_for_outbound_sockets=127.3.75.1
--tserver_master_addrs=127.3.75.62:42685
--webserver_port=40983
--webserver_interface=127.3.75.1
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab
--principal=kudu/127.3.75.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:33405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.75.20:36891/kms with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 01:01:58.992786 16144 log.cc:824] T 00000000000000000000000000000000 P e025374659ce4c9c843075e6ce79ee28: Log is configured to *not* fsync() on all Append() calls
I20260926 01:01:58.994724 16144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e025374659ce4c9c843075e6ce79ee28: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 01:01:58.995050 16144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e025374659ce4c9c843075e6ce79ee28: Bootstrap complete.
I20260926 01:01:58.997320 16144 raft_consensus.cc:348] T 00000000000000000000000000000000 P e025374659ce4c9c843075e6ce79ee28 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e025374659ce4c9c843075e6ce79ee28" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42685 } }
I20260926 01:01:58.997575 16144 raft_consensus.cc:729] T 00000000000000000000000000000000 P e025374659ce4c9c843075e6ce79ee28 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e025374659ce4c9c843075e6ce79ee28, State: Initialized, Role: FOLLOWER
I20260926 01:01:58.997737 16144 consensus_queue.cc:260] T 00000000000000000000000000000000 P e025374659ce4c9c843075e6ce79ee28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e025374659ce4c9c843075e6ce79ee28" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42685 } }
I20260926 01:01:58.997815 16144 raft_consensus.cc:388] T 00000000000000000000000000000000 P e025374659ce4c9c843075e6ce79ee28 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:01:58.997865 16144 raft_consensus.cc:482] T 00000000000000000000000000000000 P e025374659ce4c9c843075e6ce79ee28 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:01:58.997910 16144 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e025374659ce4c9c843075e6ce79ee28 [term 1 FOLLOWER]: Advancing to term 2
I20260926 01:01:58.998929 16144 raft_consensus.cc:504] T 00000000000000000000000000000000 P e025374659ce4c9c843075e6ce79ee28 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e025374659ce4c9c843075e6ce79ee28" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42685 } }
I20260926 01:01:58.999045 16144 leader_election.cc:302] T 00000000000000000000000000000000 P e025374659ce4c9c843075e6ce79ee28 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e025374659ce4c9c843075e6ce79ee28; no voters:
I20260926 01:01:58.999233 16144 leader_election.cc:288] T 00000000000000000000000000000000 P e025374659ce4c9c843075e6ce79ee28 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 01:01:58.999264 16148 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e025374659ce4c9c843075e6ce79ee28 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 01:01:58.999369 16148 raft_consensus.cc:686] T 00000000000000000000000000000000 P e025374659ce4c9c843075e6ce79ee28 [term 2 LEADER]: Becoming Leader. State: Replica: e025374659ce4c9c843075e6ce79ee28, State: Running, Role: LEADER
I20260926 01:01:58.999459 16148 consensus_queue.cc:237] T 00000000000000000000000000000000 P e025374659ce4c9c843075e6ce79ee28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e025374659ce4c9c843075e6ce79ee28" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42685 } }
I20260926 01:01:58.999504 16144 sys_catalog.cc:564] T 00000000000000000000000000000000 P e025374659ce4c9c843075e6ce79ee28 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:01:58.999745 16148 sys_catalog.cc:455] T 00000000000000000000000000000000 P e025374659ce4c9c843075e6ce79ee28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e025374659ce4c9c843075e6ce79ee28. Latest consensus state: current_term: 2 leader_uuid: "e025374659ce4c9c843075e6ce79ee28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e025374659ce4c9c843075e6ce79ee28" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42685 } } }
I20260926 01:01:58.999816 16148 sys_catalog.cc:458] T 00000000000000000000000000000000 P e025374659ce4c9c843075e6ce79ee28 [sys.catalog]: This master's current role is: LEADER
I20260926 01:01:58.999751 16149 sys_catalog.cc:455] T 00000000000000000000000000000000 P e025374659ce4c9c843075e6ce79ee28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e025374659ce4c9c843075e6ce79ee28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e025374659ce4c9c843075e6ce79ee28" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42685 } } }
I20260926 01:01:58.999969 16149 sys_catalog.cc:458] T 00000000000000000000000000000000 P e025374659ce4c9c843075e6ce79ee28 [sys.catalog]: This master's current role is: LEADER
I20260926 01:01:59.001271 16144 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 01:01:59.001647 16152 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:01:59.003965 16152 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:01:59.004287 16152 catalog_manager.cc:1259] Loaded cluster ID: 50d4d503066b41acbc2cf679a5247952
I20260926 01:01:59.004329 16152 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:01:59.006237 16152 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:01:59.006486 16152 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e025374659ce4c9c843075e6ce79ee28: Loaded TSK: 0
I20260926 01:01:59.006784 16152 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 01:01:59.086440 16147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:01:59.086701 16147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:01:59.086740 16147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:01:59.088471 16147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:01:59.088536 16147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:01:59.088575 16147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.1
I20260926 01:01:59.090818 16147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:33405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.75.20:36891/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab
--principal=kudu/127.3.75.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.1:37121
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.75.1
--webserver_port=40983
--tserver_master_addrs=127.3.75.62:42685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.1
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:01:59.091217 16147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:01:59.091509 16147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:01:59.097702 16184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:01:59.097754 16182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:01:59.097965 16181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:01:59.098191 16147 server_base.cc:1034] running on GCE node
I20260926 01:01:59.098393 16147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:01:59.098672 16147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:01:59.099830 16147 hybrid_clock.cc:648] HybridClock initialized: now 1790384519099808 us; error 36 us; skew 500 ppm
Sep 26 01:01:59 dist-test-slave-7v10 krb5kdc[15109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384519, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:01:59.103134 16147 init.cc:377] Logged in from keytab as kudu/127.3.75.1@KRBTEST.COM (short username kudu)
I20260926 01:01:59.103654 16147 webserver.cc:460] Webserver started at http://127.3.75.1:40983/ using document root <none> and password file <none>
I20260926 01:01:59.103924 16147 fs_manager.cc:359] Metadata directory not provided
I20260926 01:01:59.103996 16147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 01:01:59 dist-test-slave-7v10 krb5kdc[15109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384519, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.1@KRBTEST.COM for HTTP/127.3.75.20@KRBTEST.COM
I20260926 01:01:59.123342 16147 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:01:59.124485 16192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:01:59.124699 16147 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:01:59.124806 16147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-0/wal
uuid: "bf72ecfdc187462ca6275a2655fd74b4"
format_stamp: "Formatted at 2026-09-26 01:01:58 on dist-test-slave-7v10"
server_key: "5bfec010f648e5b1acbbb39e324352fc"
server_key_iv: "3e31ca364548e4ee2f7ffdd415bdfa01"
server_key_version: "kuduclusterkey@0"
I20260926 01:01:59.125372 16147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:01:59.147040 16147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:01:59.149361 16147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:01:59.149569 16147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:01:59.149854 16147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:01:59.150285 16147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:01:59.150380 16147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:01:59.150445 16147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:01:59.150491 16147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:01:59.164189 16147 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.1:37121
I20260926 01:01:59.164728 16147 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
I20260926 01:01:59.165560 16305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.1:37121 every 8 connection(s)
I20260926 01:01:59.169344 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 16147
I20260926 01:01:59.169687 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.75.20:36891/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.2:46241
--local_ip_for_outbound_sockets=127.3.75.2
--tserver_master_addrs=127.3.75.62:42685
--webserver_port=32899
--webserver_interface=127.3.75.2
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab
--principal=kudu/127.3.75.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:33405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.75.20:36891/kms with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 01:01:59 dist-test-slave-7v10 krb5kdc[15109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384519, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.1@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 01:01:59.180433 16308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:01:59.165768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:42685 (local address 127.3.75.1:49635)
0926 01:01:59.166144 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0926 01:01:59.166191 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 01:01:59.168273 (+ 2082us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:01:59.168908 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:01:59.168912 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:01:59.169147 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:01:59.169416 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:01:59.169423 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:01:59.171022 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:01:59.171023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:01:59.172347 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:01:59.172350 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:01:59.173327 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:01:59.173328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:01:59.173468 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:01:59.174071 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:01:59.174081 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:01:59.175926 (+ 1845us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:01:59.178335 (+ 2409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:01:59.178341 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:01:59.178344 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:01:59.178672 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:01:59.179018 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:01:59.179019 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:01:59.179020 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:01:59.179167 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:01:59.179714 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:01:59.179721 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:01:59.179907 (+ 186us) client_negotiation.cc:769] Sending connection context
0926 01:01:59.180018 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 01:01:59.180174 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":83,"threads_started":1}
I20260926 01:01:59.181293 16309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:01:59.165837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:49635 (local address 127.3.75.62:42685)
0926 01:01:59.166053 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 01:01:59.166065 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 01:01:59.166607 (+ 542us) server_negotiation.cc:407] Connection header received
0926 01:01:59.168530 (+ 1923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:01:59.168535 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:01:59.168580 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:01:59.168728 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:01:59.169556 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:01:59.170833 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:01:59.172445 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:01:59.173106 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:01:59.173212 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:01:59.176066 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:01:59.176072 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:01:59.176072 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:01:59.176086 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:01:59.178233 (+ 2147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:01:59.178776 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:01:59.178779 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:01:59.178780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:01:59.178933 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:01:59.179245 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:01:59.179246 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:01:59.179247 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:01:59.179591 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:01:59.179681 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 01:01:59.180906 (+ 1225us) server_negotiation.cc:299] Negotiation successful
0926 01:01:59.181045 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":87,"threads_started":1}
I20260926 01:01:59.182587 16306 heartbeater.cc:347] Connected to a master server at 127.3.75.62:42685
I20260926 01:01:59.182682 16306 heartbeater.cc:464] Registering TS with master...
I20260926 01:01:59.182940 16306 heartbeater.cc:511] Master 127.3.75.62:42685 requested a full tablet report, sending...
W20260926 01:01:59.183876 16306 heartbeater.cc:644] Failed to heartbeat to 127.3.75.62:42685 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 01:01:59.183943 16306 heartbeater.cc:464] Registering TS with master...
I20260926 01:01:59.184000 16306 heartbeater.cc:511] Master 127.3.75.62:42685 requested a full tablet report, sending...
I20260926 01:01:59.184312 16306 heartbeater.cc:464] Registering TS with master...
I20260926 01:01:59.184381 16306 heartbeater.cc:511] Master 127.3.75.62:42685 requested a full tablet report, sending...
W20260926 01:01:59.184659 16306 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 01:01:59.273190 16310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:01:59.273479 16310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:01:59.273530 16310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:01:59.275161 16310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:01:59.275270 16310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:01:59.275347 16310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.2
I20260926 01:01:59.277529 16310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:33405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.75.20:36891/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab
--principal=kudu/127.3.75.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.2:46241
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.75.2
--webserver_port=32899
--tserver_master_addrs=127.3.75.62:42685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.2
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:01:59.277958 16310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:01:59.278249 16310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:01:59.283609 16315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:01:59.283874 16316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:01:59.284107 16318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:01:59.285478 16310 server_base.cc:1034] running on GCE node
I20260926 01:01:59.285699 16310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:01:59.285981 16310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:01:59.287127 16310 hybrid_clock.cc:648] HybridClock initialized: now 1790384519287100 us; error 39 us; skew 500 ppm
Sep 26 01:01:59 dist-test-slave-7v10 krb5kdc[15109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384519, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:01:59.290288 16310 init.cc:377] Logged in from keytab as kudu/127.3.75.2@KRBTEST.COM (short username kudu)
I20260926 01:01:59.290751 16310 webserver.cc:460] Webserver started at http://127.3.75.2:32899/ using document root <none> and password file <none>
I20260926 01:01:59.291046 16310 fs_manager.cc:359] Metadata directory not provided
I20260926 01:01:59.291112 16310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 01:01:59 dist-test-slave-7v10 krb5kdc[15109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384519, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.2@KRBTEST.COM for HTTP/127.3.75.20@KRBTEST.COM
I20260926 01:01:59.314105 16310 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260926 01:01:59.314929 16326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:01:59.315096 16310 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:01:59.315261 16310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-1/wal
uuid: "4bd73316892d4aa7b951b06f3551e963"
format_stamp: "Formatted at 2026-09-26 01:01:58 on dist-test-slave-7v10"
server_key: "f110abdec298efd821657cf85731f25d"
server_key_iv: "00e8def8d9098142dae0ab69dbd0d6a8"
server_key_version: "kuduclusterkey@0"
I20260926 01:01:59.316413 16310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:01:59.327116 16310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:01:59.329121 16310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:01:59.329312 16310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:01:59.329586 16310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:01:59.330006 16310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:01:59.330094 16310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:01:59.330142 16310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:01:59.330160 16310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:01:59.337296 16310 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.2:46241
I20260926 01:01:59.337800 16310 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
I20260926 01:01:59.338842 16439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.2:46241 every 8 connection(s)
Sep 26 01:01:59 dist-test-slave-7v10 krb5kdc[15109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384519, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.2@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 01:01:59.346551 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 16310
I20260926 01:01:59.346837 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.75.20:36891/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.3:35935
--local_ip_for_outbound_sockets=127.3.75.3
--tserver_master_addrs=127.3.75.62:42685
--webserver_port=45367
--webserver_interface=127.3.75.3
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab
--principal=kudu/127.3.75.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:33405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.75.20:36891/kms with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 01:01:59.347368 16309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:01:59.338609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:53127 (local address 127.3.75.62:42685)
0926 01:01:59.338655 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 01:01:59.338657 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:01:59.338941 (+ 284us) server_negotiation.cc:407] Connection header received
0926 01:01:59.339635 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:01:59.339636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:01:59.339649 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:01:59.339732 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:01:59.340304 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:01:59.340802 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:01:59.341959 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:01:59.342206 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:01:59.342284 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:01:59.344411 (+ 2127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:01:59.344413 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:01:59.344413 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:01:59.344421 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:01:59.345823 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:01:59.346197 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:01:59.346198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:01:59.346198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:01:59.346254 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:01:59.346534 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:01:59.346535 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:01:59.346536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:01:59.346672 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:01:59.346720 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 01:01:59.347133 (+ 413us) server_negotiation.cc:299] Negotiation successful
0926 01:01:59.347228 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 01:01:59.347888 16442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:01:59.338591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:42685 (local address 127.3.75.2:53127)
0926 01:01:59.338758 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 01:01:59.338814 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 01:01:59.339494 (+ 680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:01:59.339843 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:01:59.339847 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:01:59.340025 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:01:59.340201 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:01:59.340205 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:01:59.340895 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:01:59.340896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:01:59.341886 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:01:59.341888 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:01:59.342315 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:01:59.342316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:01:59.342455 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:01:59.342954 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:01:59.342962 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:01:59.344317 (+ 1355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:01:59.345900 (+ 1583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:01:59.345906 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:01:59.345908 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:01:59.346115 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:01:59.346313 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:01:59.346314 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:01:59.346314 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:01:59.346444 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:01:59.346770 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:01:59.346776 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:01:59.346983 (+ 207us) client_negotiation.cc:769] Sending connection context
0926 01:01:59.347458 (+ 475us) client_negotiation.cc:240] Negotiation successful
0926 01:01:59.347618 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":94,"threads_started":1}
I20260926 01:01:59.348954 16440 heartbeater.cc:347] Connected to a master server at 127.3.75.62:42685
I20260926 01:01:59.349050 16440 heartbeater.cc:464] Registering TS with master...
I20260926 01:01:59.349229 16440 heartbeater.cc:511] Master 127.3.75.62:42685 requested a full tablet report, sending...
W20260926 01:01:59.349929 16440 heartbeater.cc:644] Failed to heartbeat to 127.3.75.62:42685 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 01:01:59.349997 16440 heartbeater.cc:464] Registering TS with master...
I20260926 01:01:59.350056 16440 heartbeater.cc:511] Master 127.3.75.62:42685 requested a full tablet report, sending...
I20260926 01:01:59.350482 16440 heartbeater.cc:464] Registering TS with master...
I20260926 01:01:59.350572 16440 heartbeater.cc:511] Master 127.3.75.62:42685 requested a full tablet report, sending...
W20260926 01:01:59.350950 16440 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 01:01:59.442742 16443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:01:59.443113 16443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:01:59.443162 16443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:01:59.444881 16443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:01:59.444975 16443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:01:59.445045 16443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.3
I20260926 01:01:59.447352 16443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:33405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.75.20:36891/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab
--principal=kudu/127.3.75.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.3:35935
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.75.3
--webserver_port=45367
--tserver_master_addrs=127.3.75.62:42685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.3
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:01:59.447759 16443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:01:59.448051 16443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:01:59.454526 16449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:01:59.454576 16451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:01:59.455231 16443 server_base.cc:1034] running on GCE node
W20260926 01:01:59.455442 16448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:01:59.455766 16443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:01:59.456036 16443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:01:59.457202 16443 hybrid_clock.cc:648] HybridClock initialized: now 1790384519457184 us; error 34 us; skew 500 ppm
Sep 26 01:01:59 dist-test-slave-7v10 krb5kdc[15109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384519, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:01:59.460464 16443 init.cc:377] Logged in from keytab as kudu/127.3.75.3@KRBTEST.COM (short username kudu)
I20260926 01:01:59.461002 16443 webserver.cc:460] Webserver started at http://127.3.75.3:45367/ using document root <none> and password file <none>
I20260926 01:01:59.461256 16443 fs_manager.cc:359] Metadata directory not provided
I20260926 01:01:59.461328 16443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 01:01:59 dist-test-slave-7v10 krb5kdc[15109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384519, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.3@KRBTEST.COM for HTTP/127.3.75.20@KRBTEST.COM
I20260926 01:01:59.484484 16443 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:01:59.494145 16460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:01:59.494338 16443 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.001s sys 0.000s
I20260926 01:01:59.494453 16443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-2/wal
uuid: "25cbfa91103e44df90817bd7ad62cb45"
format_stamp: "Formatted at 2026-09-26 01:01:58 on dist-test-slave-7v10"
server_key: "5b3d6c8ca5ee5b190ca377f28fafc8d5"
server_key_iv: "c9621477c9c7d6b0b49ecbf034cf58b3"
server_key_version: "kuduclusterkey@0"
I20260926 01:01:59.495023 16443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:01:59.502192 16443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:01:59.504266 16443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:01:59.504428 16443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:01:59.504648 16443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:01:59.505008 16443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:01:59.505051 16443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:01:59.505118 16443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:01:59.505147 16443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:01:59.515252 16443 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.3:35935
I20260926 01:01:59.515642 16443 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
I20260926 01:01:59.515960 16574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.3:35935 every 8 connection(s)
Sep 26 01:01:59 dist-test-slave-7v10 krb5kdc[15109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384519, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.3@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 01:01:59.524178 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 16443
I20260926 01:01:59.525786 16309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:01:59.516957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.3:40543 (local address 127.3.75.62:42685)
0926 01:01:59.517005 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 01:01:59.517007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:01:59.517017 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:01:59.517247 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:01:59.517249 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:01:59.517260 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:01:59.517315 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:01:59.518014 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:01:59.518766 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:01:59.520363 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:01:59.520659 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:01:59.520707 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:01:59.523011 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:01:59.523013 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:01:59.523014 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:01:59.523021 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:01:59.524385 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:01:59.524849 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:01:59.524850 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:01:59.524851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:01:59.524915 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:01:59.525159 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:01:59.525160 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:01:59.525161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:01:59.525273 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:01:59.525346 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 01:01:59.525621 (+ 275us) server_negotiation.cc:299] Negotiation successful
0926 01:01:59.525701 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 01:01:59.525985 16577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:01:59.516410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:42685 (local address 127.3.75.3:40543)
0926 01:01:59.516565 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 01:01:59.516609 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 01:01:59.517149 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:01:59.517416 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:01:59.517421 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:01:59.517619 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:01:59.517889 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:01:59.517896 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:01:59.518925 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:01:59.518926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:01:59.520256 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:01:59.520259 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:01:59.520818 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:01:59.520819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:01:59.521007 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:01:59.521617 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:01:59.521625 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:01:59.522933 (+ 1308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:01:59.524474 (+ 1541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:01:59.524480 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:01:59.524482 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:01:59.524773 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:01:59.524984 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:01:59.524985 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:01:59.524985 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:01:59.525095 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:01:59.525336 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:01:59.525342 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:01:59.525483 (+ 141us) client_negotiation.cc:769] Sending connection context
0926 01:01:59.525599 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 01:01:59.525733 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":98,"threads_started":1}
I20260926 01:01:59.526507 16575 heartbeater.cc:347] Connected to a master server at 127.3.75.62:42685
I20260926 01:01:59.526623 16575 heartbeater.cc:464] Registering TS with master...
I20260926 01:01:59.526794 16575 heartbeater.cc:511] Master 127.3.75.62:42685 requested a full tablet report, sending...
W20260926 01:01:59.527403 16575 heartbeater.cc:644] Failed to heartbeat to 127.3.75.62:42685 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 01:01:59.527479 16575 heartbeater.cc:464] Registering TS with master...
I20260926 01:01:59.527549 16575 heartbeater.cc:511] Master 127.3.75.62:42685 requested a full tablet report, sending...
I20260926 01:01:59.527814 16575 heartbeater.cc:464] Registering TS with master...
I20260926 01:01:59.527928 16575 heartbeater.cc:511] Master 127.3.75.62:42685 requested a full tablet report, sending...
W20260926 01:01:59.528190 16575 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 01:01:59.534626 16578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:01:59.524579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40830 (local address 127.3.75.62:42685)
0926 01:01:59.525148 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0926 01:01:59.525150 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:01:59.525323 (+ 173us) server_negotiation.cc:407] Connection header received
0926 01:01:59.525424 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:01:59.525425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:01:59.525438 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:01:59.525476 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:01:59.527034 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:01:59.527767 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:01:59.528952 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:01:59.529256 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:01:59.529316 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:01:59.530962 (+ 1646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:01:59.530964 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:01:59.530964 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:01:59.530971 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:01:59.532657 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:01:59.533745 (+ 1088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:01:59.533746 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:01:59.533746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:01:59.533812 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:01:59.534052 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:01:59.534053 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:01:59.534053 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:01:59.534188 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:01:59.534370 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0926 01:01:59.534437 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 01:01:59.534523 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":55,"threads_started":1}
Sep 26 01:02:00 dist-test-slave-7v10 krb5kdc[15109](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790384520, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:02:00.184831 16306 heartbeater.cc:464] Registering TS with master...
I20260926 01:02:00.184996 16306 heartbeater.cc:511] Master 127.3.75.62:42685 requested a full tablet report, sending...
I20260926 01:02:00.185400 16108 ts_manager.cc:194] Registered new tserver with Master: bf72ecfdc187462ca6275a2655fd74b4 (127.3.75.1:37121)
I20260926 01:02:00.186372 16108 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.75.1@KRBTEST.COM'} at 127.3.75.1:49635
I20260926 01:02:00.351130 16440 heartbeater.cc:464] Registering TS with master...
I20260926 01:02:00.351284 16440 heartbeater.cc:511] Master 127.3.75.62:42685 requested a full tablet report, sending...
I20260926 01:02:00.351600 16108 ts_manager.cc:194] Registered new tserver with Master: 4bd73316892d4aa7b951b06f3551e963 (127.3.75.2:46241)
I20260926 01:02:00.352165 16108 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.75.2@KRBTEST.COM'} at 127.3.75.2:53127
Sep 26 01:02:00 dist-test-slave-7v10 krb5kdc[15109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384520, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.62@KRBTEST.COM for HTTP/127.3.75.20@KRBTEST.COM
I20260926 01:02:00.528376 16575 heartbeater.cc:464] Registering TS with master...
I20260926 01:02:00.528548 16575 heartbeater.cc:511] Master 127.3.75.62:42685 requested a full tablet report, sending...
I20260926 01:02:00.528905 16108 ts_manager.cc:194] Registered new tserver with Master: 25cbfa91103e44df90817bd7ad62cb45 (127.3.75.3:35935)
I20260926 01:02:00.529456 16108 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.75.3@KRBTEST.COM'} at 127.3.75.3:40543
I20260926 01:02:00.531152 3372 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 01:02:00.548041 16598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:00.539742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40846 (local address 127.3.75.62:42685)
0926 01:02:00.539878 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:00.539880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:00.539985 (+ 105us) server_negotiation.cc:407] Connection header received
0926 01:02:00.540045 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:00.540045 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:00.540060 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:00.540156 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:02:00.540893 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:00.541694 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:00.542782 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:00.543135 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:00.543220 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:00.544722 (+ 1502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:02:00.544724 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:02:00.544725 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:02:00.544733 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:02:00.546694 (+ 1961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:00.547136 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:00.547137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:00.547137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:00.547238 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:00.547458 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:00.547459 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:00.547460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:00.547607 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:02:00.547715 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:00.547833 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 01:02:00.547920 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":75,"threads_started":1}
I20260926 01:02:00.550429 16108 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40846:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:02:00.552773 16108 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:02:00.563833 16604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:00.556030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.3:35935 (local address 127.0.0.1:59516)
0926 01:02:00.559091 (+ 3061us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:00.559143 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 01:02:00.559217 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:02:00.559831 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:02:00.559832 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:02:00.559835 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:02:00.559914 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:00.559916 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:00.561327 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:00.561328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:00.562543 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:00.562546 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:00.563553 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:00.563554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:00.563644 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:00.563659 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 01:02:00.563706 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 01:02:00.563756 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3028,"thread_start_us":82,"threads_started":1}
I20260926 01:02:00.564275 16607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:00.556304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59516 (local address 127.3.75.3:35935)
0926 01:02:00.559657 (+ 3353us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:00.559662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:00.559673 (+ 11us) server_negotiation.cc:407] Connection header received
0926 01:02:00.559711 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:00.559713 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:00.559729 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:00.559997 (+ 268us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:02:00.560089 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:00.561228 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:00.562651 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:00.563452 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:00.564067 (+ 615us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:00.564135 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:00.564160 (+ 25us) server_negotiation.cc:299] Negotiation successful
0926 01:02:00.564200 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3311,"thread_start_us":62,"threads_started":1}
I20260926 01:02:00.565246 16506 tablet_service.cc:1467] Processing CreateTablet for tablet 9aa8d4c962894f27b2ac457a5174eafc (DEFAULT_TABLE table=test-table [id=3d634fdf262a4eec8ecf6c5ce84df189]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:02:00.565608 16506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aa8d4c962894f27b2ac457a5174eafc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:02:00.567526 16603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:00.556037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.1:37121 (local address 127.0.0.1:38230)
0926 01:02:00.559336 (+ 3299us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:00.559346 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:02:00.559398 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:02:00.559727 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:02:00.559730 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:02:00.559733 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:02:00.559805 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:00.559808 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:00.563040 (+ 3232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:00.563041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:00.564826 (+ 1785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:00.564829 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:00.567297 (+ 2468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:00.567298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:00.567354 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:00.567358 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:02:00.567411 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 01:02:00.567448 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3283,"thread_start_us":107,"threads_started":1}
I20260926 01:02:00.568117 16605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:00.556135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38230 (local address 127.3.75.1:37121)
0926 01:02:00.559009 (+ 2874us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:00.559015 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:00.559424 (+ 409us) server_negotiation.cc:407] Connection header received
0926 01:02:00.559484 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:00.559486 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:00.559506 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:00.559604 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:02:00.559902 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:00.561025 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:00.564928 (+ 3903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:00.565803 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:00.565879 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:00.565930 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:00.567982 (+ 2052us) server_negotiation.cc:299] Negotiation successful
0926 01:02:00.568032 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2830,"thread_start_us":50,"threads_started":1}
I20260926 01:02:00.569020 16221 tablet_service.cc:1467] Processing CreateTablet for tablet 9aa8d4c962894f27b2ac457a5174eafc (DEFAULT_TABLE table=test-table [id=3d634fdf262a4eec8ecf6c5ce84df189]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:02:00.569327 16221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aa8d4c962894f27b2ac457a5174eafc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:02:00.569397 16606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:00.556284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.2:46241 (local address 127.0.0.1:49160)
0926 01:02:00.559485 (+ 3201us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:00.559494 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:02:00.559537 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:02:00.561618 (+ 2081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:02:00.561619 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:02:00.561622 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:02:00.561666 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:00.561668 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:00.565996 (+ 4328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:00.565997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:00.567215 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:00.567218 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:00.569167 (+ 1949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:00.569168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:00.569247 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:00.569251 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:02:00.569285 (+ 34us) client_negotiation.cc:240] Negotiation successful
0926 01:02:00.569322 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3186,"thread_start_us":67,"threads_started":1}
I20260926 01:02:00.569801 16608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:00.556499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49160 (local address 127.3.75.2:46241)
0926 01:02:00.556657 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:00.556663 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:00.561441 (+ 4778us) server_negotiation.cc:407] Connection header received
0926 01:02:00.561488 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:00.561490 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:00.561508 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:00.561748 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:02:00.561840 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:00.562943 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:00.568272 (+ 5329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:00.569033 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:00.569565 (+ 532us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:00.569639 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:00.569668 (+ 29us) server_negotiation.cc:299] Negotiation successful
0926 01:02:00.569714 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":81,"threads_started":1}
I20260926 01:02:00.570539 16374 tablet_service.cc:1467] Processing CreateTablet for tablet 9aa8d4c962894f27b2ac457a5174eafc (DEFAULT_TABLE table=test-table [id=3d634fdf262a4eec8ecf6c5ce84df189]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:02:00.570852 16374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aa8d4c962894f27b2ac457a5174eafc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:02:00.572752 16609 tablet_bootstrap.cc:492] T 9aa8d4c962894f27b2ac457a5174eafc P 25cbfa91103e44df90817bd7ad62cb45: Bootstrap starting.
I20260926 01:02:00.573349 16609 tablet_bootstrap.cc:654] T 9aa8d4c962894f27b2ac457a5174eafc P 25cbfa91103e44df90817bd7ad62cb45: Neither blocks nor log segments found. Creating new log.
I20260926 01:02:00.573655 16609 log.cc:824] T 9aa8d4c962894f27b2ac457a5174eafc P 25cbfa91103e44df90817bd7ad62cb45: Log is configured to *not* fsync() on all Append() calls
I20260926 01:02:00.574397 16610 tablet_bootstrap.cc:492] T 9aa8d4c962894f27b2ac457a5174eafc P bf72ecfdc187462ca6275a2655fd74b4: Bootstrap starting.
I20260926 01:02:00.575013 16610 tablet_bootstrap.cc:654] T 9aa8d4c962894f27b2ac457a5174eafc P bf72ecfdc187462ca6275a2655fd74b4: Neither blocks nor log segments found. Creating new log.
I20260926 01:02:00.575304 16612 tablet_bootstrap.cc:492] T 9aa8d4c962894f27b2ac457a5174eafc P 4bd73316892d4aa7b951b06f3551e963: Bootstrap starting.
I20260926 01:02:00.575519 16609 tablet_bootstrap.cc:492] T 9aa8d4c962894f27b2ac457a5174eafc P 25cbfa91103e44df90817bd7ad62cb45: No bootstrap required, opened a new log
I20260926 01:02:00.575582 16609 ts_tablet_manager.cc:1397] T 9aa8d4c962894f27b2ac457a5174eafc P 25cbfa91103e44df90817bd7ad62cb45: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 01:02:00.575939 16612 tablet_bootstrap.cc:654] T 9aa8d4c962894f27b2ac457a5174eafc P 4bd73316892d4aa7b951b06f3551e963: Neither blocks nor log segments found. Creating new log.
I20260926 01:02:00.576246 16612 log.cc:824] T 9aa8d4c962894f27b2ac457a5174eafc P 4bd73316892d4aa7b951b06f3551e963: Log is configured to *not* fsync() on all Append() calls
I20260926 01:02:00.576930 16612 tablet_bootstrap.cc:492] T 9aa8d4c962894f27b2ac457a5174eafc P 4bd73316892d4aa7b951b06f3551e963: No bootstrap required, opened a new log
I20260926 01:02:00.576987 16612 ts_tablet_manager.cc:1397] T 9aa8d4c962894f27b2ac457a5174eafc P 4bd73316892d4aa7b951b06f3551e963: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:02:00.577992 16609 raft_consensus.cc:348] T 9aa8d4c962894f27b2ac457a5174eafc P 25cbfa91103e44df90817bd7ad62cb45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25cbfa91103e44df90817bd7ad62cb45" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 35935 } } peers { permanent_uuid: "4bd73316892d4aa7b951b06f3551e963" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 46241 } } peers { permanent_uuid: "bf72ecfdc187462ca6275a2655fd74b4" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37121 } }
I20260926 01:02:00.578138 16609 raft_consensus.cc:374] T 9aa8d4c962894f27b2ac457a5174eafc P 25cbfa91103e44df90817bd7ad62cb45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:02:00.578169 16609 raft_consensus.cc:729] T 9aa8d4c962894f27b2ac457a5174eafc P 25cbfa91103e44df90817bd7ad62cb45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25cbfa91103e44df90817bd7ad62cb45, State: Initialized, Role: FOLLOWER
I20260926 01:02:00.578454 16609 consensus_queue.cc:260] T 9aa8d4c962894f27b2ac457a5174eafc P 25cbfa91103e44df90817bd7ad62cb45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25cbfa91103e44df90817bd7ad62cb45" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 35935 } } peers { permanent_uuid: "4bd73316892d4aa7b951b06f3551e963" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 46241 } } peers { permanent_uuid: "bf72ecfdc187462ca6275a2655fd74b4" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37121 } }
I20260926 01:02:00.578789 16609 ts_tablet_manager.cc:1428] T 9aa8d4c962894f27b2ac457a5174eafc P 25cbfa91103e44df90817bd7ad62cb45: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 01:02:00.578982 16575 heartbeater.cc:503] Master 127.3.75.62:42685 was elected leader, sending a full tablet report...
I20260926 01:02:00.579504 16612 raft_consensus.cc:348] T 9aa8d4c962894f27b2ac457a5174eafc P 4bd73316892d4aa7b951b06f3551e963 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25cbfa91103e44df90817bd7ad62cb45" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 35935 } } peers { permanent_uuid: "4bd73316892d4aa7b951b06f3551e963" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 46241 } } peers { permanent_uuid: "bf72ecfdc187462ca6275a2655fd74b4" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37121 } }
I20260926 01:02:00.579653 16612 raft_consensus.cc:374] T 9aa8d4c962894f27b2ac457a5174eafc P 4bd73316892d4aa7b951b06f3551e963 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:02:00.579685 16612 raft_consensus.cc:729] T 9aa8d4c962894f27b2ac457a5174eafc P 4bd73316892d4aa7b951b06f3551e963 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bd73316892d4aa7b951b06f3551e963, State: Initialized, Role: FOLLOWER
I20260926 01:02:00.580185 16612 consensus_queue.cc:260] T 9aa8d4c962894f27b2ac457a5174eafc P 4bd73316892d4aa7b951b06f3551e963 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25cbfa91103e44df90817bd7ad62cb45" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 35935 } } peers { permanent_uuid: "4bd73316892d4aa7b951b06f3551e963" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 46241 } } peers { permanent_uuid: "bf72ecfdc187462ca6275a2655fd74b4" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37121 } }
I20260926 01:02:00.580264 16610 log.cc:824] T 9aa8d4c962894f27b2ac457a5174eafc P bf72ecfdc187462ca6275a2655fd74b4: Log is configured to *not* fsync() on all Append() calls
I20260926 01:02:00.580474 16612 ts_tablet_manager.cc:1428] T 9aa8d4c962894f27b2ac457a5174eafc P 4bd73316892d4aa7b951b06f3551e963: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 01:02:00.580593 16440 heartbeater.cc:503] Master 127.3.75.62:42685 was elected leader, sending a full tablet report...
I20260926 01:02:00.581748 16610 tablet_bootstrap.cc:492] T 9aa8d4c962894f27b2ac457a5174eafc P bf72ecfdc187462ca6275a2655fd74b4: No bootstrap required, opened a new log
I20260926 01:02:00.581820 16610 ts_tablet_manager.cc:1397] T 9aa8d4c962894f27b2ac457a5174eafc P bf72ecfdc187462ca6275a2655fd74b4: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20260926 01:02:00.583954 16610 raft_consensus.cc:348] T 9aa8d4c962894f27b2ac457a5174eafc P bf72ecfdc187462ca6275a2655fd74b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25cbfa91103e44df90817bd7ad62cb45" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 35935 } } peers { permanent_uuid: "4bd73316892d4aa7b951b06f3551e963" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 46241 } } peers { permanent_uuid: "bf72ecfdc187462ca6275a2655fd74b4" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37121 } }
I20260926 01:02:00.584113 16610 raft_consensus.cc:374] T 9aa8d4c962894f27b2ac457a5174eafc P bf72ecfdc187462ca6275a2655fd74b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:02:00.584160 16610 raft_consensus.cc:729] T 9aa8d4c962894f27b2ac457a5174eafc P bf72ecfdc187462ca6275a2655fd74b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf72ecfdc187462ca6275a2655fd74b4, State: Initialized, Role: FOLLOWER
I20260926 01:02:00.584268 16610 consensus_queue.cc:260] T 9aa8d4c962894f27b2ac457a5174eafc P bf72ecfdc187462ca6275a2655fd74b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25cbfa91103e44df90817bd7ad62cb45" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 35935 } } peers { permanent_uuid: "4bd73316892d4aa7b951b06f3551e963" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 46241 } } peers { permanent_uuid: "bf72ecfdc187462ca6275a2655fd74b4" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37121 } }
I20260926 01:02:00.584527 16610 ts_tablet_manager.cc:1428] T 9aa8d4c962894f27b2ac457a5174eafc P bf72ecfdc187462ca6275a2655fd74b4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 01:02:00.584580 16306 heartbeater.cc:503] Master 127.3.75.62:42685 was elected leader, sending a full tablet report...
W20260926 01:02:00.589876 16441 tablet.cc:2369] T 9aa8d4c962894f27b2ac457a5174eafc P 4bd73316892d4aa7b951b06f3551e963: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:02:00.593544 16617 raft_consensus.cc:482] T 9aa8d4c962894f27b2ac457a5174eafc P bf72ecfdc187462ca6275a2655fd74b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:02:00.593640 16617 raft_consensus.cc:504] T 9aa8d4c962894f27b2ac457a5174eafc P bf72ecfdc187462ca6275a2655fd74b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25cbfa91103e44df90817bd7ad62cb45" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 35935 } } peers { permanent_uuid: "4bd73316892d4aa7b951b06f3551e963" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 46241 } } peers { permanent_uuid: "bf72ecfdc187462ca6275a2655fd74b4" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37121 } }
I20260926 01:02:00.594595 16617 leader_election.cc:288] T 9aa8d4c962894f27b2ac457a5174eafc P bf72ecfdc187462ca6275a2655fd74b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 25cbfa91103e44df90817bd7ad62cb45 (127.3.75.3:35935), 4bd73316892d4aa7b951b06f3551e963 (127.3.75.2:46241)
I20260926 01:02:00.598213 16608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:00.594296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:41979 (local address 127.3.75.2:46241)
0926 01:02:00.594334 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:00.594336 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:00.594532 (+ 196us) server_negotiation.cc:407] Connection header received
0926 01:02:00.594781 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:00.594782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:00.594795 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:00.594852 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:02:00.595065 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:00.595784 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:00.597181 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:00.597782 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:00.597829 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:00.597839 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:00.598104 (+ 265us) server_negotiation.cc:299] Negotiation successful
0926 01:02:00.598148 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 01:02:00.598297 16607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:00.594353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:44483 (local address 127.3.75.3:35935)
0926 01:02:00.594395 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:00.594397 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:00.594630 (+ 233us) server_negotiation.cc:407] Connection header received
0926 01:02:00.594812 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:00.594813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:00.594824 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:00.594856 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:02:00.595112 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:00.595846 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:00.597279 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:00.597869 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:00.597918 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:00.597927 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:00.598181 (+ 254us) server_negotiation.cc:299] Negotiation successful
0926 01:02:00.598230 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 01:02:00.598572 16619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:00.594118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.3:35935 (local address 127.3.75.1:44483)
0926 01:02:00.594578 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:00.594587 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:02:00.594656 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:02:00.594955 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:02:00.594956 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:02:00.594958 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:02:00.595031 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:00.595033 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:00.595974 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:00.595975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:00.597152 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:00.597155 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:00.598014 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:00.598015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:00.598095 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:00.598099 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:02:00.598429 (+ 330us) client_negotiation.cc:240] Negotiation successful
0926 01:02:00.598471 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":113,"threads_started":1}
I20260926 01:02:00.598670 16618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:00.594106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.2:46241 (local address 127.3.75.1:41979)
0926 01:02:00.594463 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:00.594475 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:02:00.594708 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:02:00.594916 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:02:00.594917 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:02:00.594919 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:02:00.594990 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:00.594992 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:00.595886 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:00.595887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:00.597072 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:00.597075 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:00.597944 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:00.597945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:00.598020 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:00.598024 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:02:00.598542 (+ 518us) client_negotiation.cc:240] Negotiation successful
0926 01:02:00.598589 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":73,"threads_started":1}
I20260926 01:02:00.599103 16525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa8d4c962894f27b2ac457a5174eafc" candidate_uuid: "bf72ecfdc187462ca6275a2655fd74b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25cbfa91103e44df90817bd7ad62cb45" is_pre_election: true
I20260926 01:02:00.599270 16525 raft_consensus.cc:2454] T 9aa8d4c962894f27b2ac457a5174eafc P 25cbfa91103e44df90817bd7ad62cb45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf72ecfdc187462ca6275a2655fd74b4 in term 0.
I20260926 01:02:00.599262 16394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa8d4c962894f27b2ac457a5174eafc" candidate_uuid: "bf72ecfdc187462ca6275a2655fd74b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bd73316892d4aa7b951b06f3551e963" is_pre_election: true
I20260926 01:02:00.599393 16394 raft_consensus.cc:2454] T 9aa8d4c962894f27b2ac457a5174eafc P 4bd73316892d4aa7b951b06f3551e963 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf72ecfdc187462ca6275a2655fd74b4 in term 0.
I20260926 01:02:00.599538 16194 leader_election.cc:302] T 9aa8d4c962894f27b2ac457a5174eafc P bf72ecfdc187462ca6275a2655fd74b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25cbfa91103e44df90817bd7ad62cb45, bf72ecfdc187462ca6275a2655fd74b4; no voters:
I20260926 01:02:00.599740 16617 raft_consensus.cc:2781] T 9aa8d4c962894f27b2ac457a5174eafc P bf72ecfdc187462ca6275a2655fd74b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:02:00.599788 16617 raft_consensus.cc:482] T 9aa8d4c962894f27b2ac457a5174eafc P bf72ecfdc187462ca6275a2655fd74b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:02:00.599814 16617 raft_consensus.cc:3037] T 9aa8d4c962894f27b2ac457a5174eafc P bf72ecfdc187462ca6275a2655fd74b4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:02:00.600775 16617 raft_consensus.cc:504] T 9aa8d4c962894f27b2ac457a5174eafc P bf72ecfdc187462ca6275a2655fd74b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25cbfa91103e44df90817bd7ad62cb45" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 35935 } } peers { permanent_uuid: "4bd73316892d4aa7b951b06f3551e963" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 46241 } } peers { permanent_uuid: "bf72ecfdc187462ca6275a2655fd74b4" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37121 } }
I20260926 01:02:00.601030 16525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa8d4c962894f27b2ac457a5174eafc" candidate_uuid: "bf72ecfdc187462ca6275a2655fd74b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25cbfa91103e44df90817bd7ad62cb45"
I20260926 01:02:00.601104 16525 raft_consensus.cc:3037] T 9aa8d4c962894f27b2ac457a5174eafc P 25cbfa91103e44df90817bd7ad62cb45 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:02:00.601130 16394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa8d4c962894f27b2ac457a5174eafc" candidate_uuid: "bf72ecfdc187462ca6275a2655fd74b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bd73316892d4aa7b951b06f3551e963"
I20260926 01:02:00.601204 16394 raft_consensus.cc:3037] T 9aa8d4c962894f27b2ac457a5174eafc P 4bd73316892d4aa7b951b06f3551e963 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:02:00.602011 16525 raft_consensus.cc:2454] T 9aa8d4c962894f27b2ac457a5174eafc P 25cbfa91103e44df90817bd7ad62cb45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf72ecfdc187462ca6275a2655fd74b4 in term 1.
I20260926 01:02:00.602033 16394 raft_consensus.cc:2454] T 9aa8d4c962894f27b2ac457a5174eafc P 4bd73316892d4aa7b951b06f3551e963 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf72ecfdc187462ca6275a2655fd74b4 in term 1.
I20260926 01:02:00.602272 16195 leader_election.cc:302] T 9aa8d4c962894f27b2ac457a5174eafc P bf72ecfdc187462ca6275a2655fd74b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bd73316892d4aa7b951b06f3551e963, bf72ecfdc187462ca6275a2655fd74b4; no voters:
I20260926 01:02:00.602469 16617 leader_election.cc:288] T 9aa8d4c962894f27b2ac457a5174eafc P bf72ecfdc187462ca6275a2655fd74b4 [CANDIDATE]: Term 1 election: Requested vote from peers 25cbfa91103e44df90817bd7ad62cb45 (127.3.75.3:35935), 4bd73316892d4aa7b951b06f3551e963 (127.3.75.2:46241)
I20260926 01:02:00.602499 16620 raft_consensus.cc:2781] T 9aa8d4c962894f27b2ac457a5174eafc P bf72ecfdc187462ca6275a2655fd74b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:02:00.602644 16620 raft_consensus.cc:686] T 9aa8d4c962894f27b2ac457a5174eafc P bf72ecfdc187462ca6275a2655fd74b4 [term 1 LEADER]: Becoming Leader. State: Replica: bf72ecfdc187462ca6275a2655fd74b4, State: Running, Role: LEADER
I20260926 01:02:00.602746 16620 consensus_queue.cc:237] T 9aa8d4c962894f27b2ac457a5174eafc P bf72ecfdc187462ca6275a2655fd74b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25cbfa91103e44df90817bd7ad62cb45" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 35935 } } peers { permanent_uuid: "4bd73316892d4aa7b951b06f3551e963" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 46241 } } peers { permanent_uuid: "bf72ecfdc187462ca6275a2655fd74b4" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37121 } }
I20260926 01:02:00.603629 16107 catalog_manager.cc:5495] T 9aa8d4c962894f27b2ac457a5174eafc P bf72ecfdc187462ca6275a2655fd74b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf72ecfdc187462ca6275a2655fd74b4 (127.3.75.1). New cstate: current_term: 1 leader_uuid: "bf72ecfdc187462ca6275a2655fd74b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25cbfa91103e44df90817bd7ad62cb45" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 35935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bd73316892d4aa7b951b06f3551e963" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 46241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf72ecfdc187462ca6275a2655fd74b4" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37121 } health_report { overall_health: HEALTHY } } }
I20260926 01:02:00.621599 16605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:00.618402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38242 (local address 127.3.75.1:37121)
0926 01:02:00.618442 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:00.618444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:00.618479 (+ 35us) server_negotiation.cc:407] Connection header received
0926 01:02:00.618511 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:00.618512 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:00.618531 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:00.618596 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:02:00.618747 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:00.619520 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:00.620637 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:00.620936 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:00.621017 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:00.621189 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:02:00.621315 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:02:00.621407 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:00.621492 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 01:02:00.621513 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 01:02:00.623483 16394 raft_consensus.cc:1261] T 9aa8d4c962894f27b2ac457a5174eafc P 4bd73316892d4aa7b951b06f3551e963 [term 1 FOLLOWER]: Refusing update from remote peer bf72ecfdc187462ca6275a2655fd74b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:02:00.623546 16525 raft_consensus.cc:1261] T 9aa8d4c962894f27b2ac457a5174eafc P 25cbfa91103e44df90817bd7ad62cb45 [term 1 FOLLOWER]: Refusing update from remote peer bf72ecfdc187462ca6275a2655fd74b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:02:00.623793 16620 consensus_queue.cc:1035] T 9aa8d4c962894f27b2ac457a5174eafc P bf72ecfdc187462ca6275a2655fd74b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bd73316892d4aa7b951b06f3551e963" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 46241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:02:00.623948 16617 consensus_queue.cc:1035] T 9aa8d4c962894f27b2ac457a5174eafc P bf72ecfdc187462ca6275a2655fd74b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25cbfa91103e44df90817bd7ad62cb45" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 35935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:02:00.627892 16625 mvcc.cc:205] Tried to move back new op lower bound from 7333414996471382016 to 7333414996389621760. Current Snapshot: MvccSnapshot[applied={T|T < 7333414996471382016}]
I20260926 01:02:00.628584 16107 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40846:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:02:00.628667 16107 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40846:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:02:00.629637 16624 mvcc.cc:205] Tried to move back new op lower bound from 7333414996471382016 to 7333414996389621760. Current Snapshot: MvccSnapshot[applied={T|T < 7333414996471382016}]
I20260926 01:02:00.629642 16107 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e025374659ce4c9c843075e6ce79ee28: Sending DeleteTablet for 3 replicas of tablet 9aa8d4c962894f27b2ac457a5174eafc
I20260926 01:02:00.629973 16221 tablet_service.cc:1514] Processing DeleteTablet for tablet 9aa8d4c962894f27b2ac457a5174eafc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:02:00 UTC) from {username='kudu', principal='kudu/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:38230
I20260926 01:02:00.630333 16506 tablet_service.cc:1514] Processing DeleteTablet for tablet 9aa8d4c962894f27b2ac457a5174eafc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:02:00 UTC) from {username='kudu', principal='kudu/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:59516
I20260926 01:02:00.630940 16374 tablet_service.cc:1514] Processing DeleteTablet for tablet 9aa8d4c962894f27b2ac457a5174eafc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:02:00 UTC) from {username='kudu', principal='kudu/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:49160
I20260926 01:02:00.631664 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 16147
I20260926 01:02:00.635002 16635 tablet_replica.cc:330] T 9aa8d4c962894f27b2ac457a5174eafc P 4bd73316892d4aa7b951b06f3551e963: stopping tablet replica
I20260926 01:02:00.635095 16635 raft_consensus.cc:2229] T 9aa8d4c962894f27b2ac457a5174eafc P 4bd73316892d4aa7b951b06f3551e963 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:02:00.635282 16634 tablet_replica.cc:330] T 9aa8d4c962894f27b2ac457a5174eafc P 25cbfa91103e44df90817bd7ad62cb45: stopping tablet replica
I20260926 01:02:00.635380 16634 raft_consensus.cc:2229] T 9aa8d4c962894f27b2ac457a5174eafc P 25cbfa91103e44df90817bd7ad62cb45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:02:00.635444 16634 raft_consensus.cc:2258] T 9aa8d4c962894f27b2ac457a5174eafc P 25cbfa91103e44df90817bd7ad62cb45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:02:00.635550 16635 raft_consensus.cc:2258] T 9aa8d4c962894f27b2ac457a5174eafc P 4bd73316892d4aa7b951b06f3551e963 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:02:00.636018 16635 ts_tablet_manager.cc:1905] T 9aa8d4c962894f27b2ac457a5174eafc P 4bd73316892d4aa7b951b06f3551e963: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:02:00.637979 16635 ts_tablet_manager.cc:1918] T 9aa8d4c962894f27b2ac457a5174eafc P 4bd73316892d4aa7b951b06f3551e963: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:02:00.638087 16635 log.cc:1196] T 9aa8d4c962894f27b2ac457a5174eafc P 4bd73316892d4aa7b951b06f3551e963: Deleting WAL directory at /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-1/wal/wals/9aa8d4c962894f27b2ac457a5174eafc
I20260926 01:02:00.638386 16635 ts_tablet_manager.cc:1939] T 9aa8d4c962894f27b2ac457a5174eafc P 4bd73316892d4aa7b951b06f3551e963: Deleting consensus metadata
I20260926 01:02:00.638790 16094 catalog_manager.cc:4841] TS 4bd73316892d4aa7b951b06f3551e963 (127.3.75.2:46241): tablet 9aa8d4c962894f27b2ac457a5174eafc (table test-table [id=3d634fdf262a4eec8ecf6c5ce84df189]) successfully deleted
I20260926 01:02:00.639344 16634 ts_tablet_manager.cc:1905] T 9aa8d4c962894f27b2ac457a5174eafc P 25cbfa91103e44df90817bd7ad62cb45: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 01:02:00.640412 16092 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.75.1:37121 (error 108)
I20260926 01:02:00.640873 16634 ts_tablet_manager.cc:1918] T 9aa8d4c962894f27b2ac457a5174eafc P 25cbfa91103e44df90817bd7ad62cb45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:02:00.640945 16634 log.cc:1196] T 9aa8d4c962894f27b2ac457a5174eafc P 25cbfa91103e44df90817bd7ad62cb45: Deleting WAL directory at /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384372073861-3372-0/minicluster-data/ts-2/wal/wals/9aa8d4c962894f27b2ac457a5174eafc
I20260926 01:02:00.641139 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 16310
I20260926 01:02:00.641215 16634 ts_tablet_manager.cc:1939] T 9aa8d4c962894f27b2ac457a5174eafc P 25cbfa91103e44df90817bd7ad62cb45: Deleting consensus metadata
I20260926 01:02:00.643321 16093 catalog_manager.cc:4841] TS 25cbfa91103e44df90817bd7ad62cb45 (127.3.75.3:35935): tablet 9aa8d4c962894f27b2ac457a5174eafc (table test-table [id=3d634fdf262a4eec8ecf6c5ce84df189]) successfully deleted
I20260926 01:02:00.647845 16606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:00.647716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.1:37121 (local address 127.0.0.1:38254)
0926 01:02:00.647764 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:00.647810 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.1:37121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260926 01:02:00.648098 16092 catalog_manager.cc:4584] TS bf72ecfdc187462ca6275a2655fd74b4 (127.3.75.1:37121): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9aa8d4c962894f27b2ac457a5174eafc: Network error: Client connection negotiation failed: client connection to 127.3.75.1:37121: connect: Connection refused (error 111)
I20260926 01:02:00.648206 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 16443
I20260926 01:02:00.653427 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 16075
I20260926 01:02:00.658814 3372 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 01:02:00.992718 3372 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 01:02:00.992817 3372 mini_ranger.cc:66] Stopping Ranger...
I20260926 01:02:01.337598 3372 mini_ranger.cc:68] Stopped Ranger
2026-09-26 01:02:01.337 UTC [15139] LOG: received smart shutdown request
2026-09-26 01:02:01.341 UTC [15139] LOG: background worker "logical replication launcher" (PID 15148) exited with exit code 1
2026-09-26 01:02:01.341 UTC [15143] LOG: shutting down
2026-09-26 01:02:01.504 UTC [15139] LOG: database system is shut down
2026-09-26T01:02:01Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (46937 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:02:01 dist-test-slave-7v10 krb5kdc[16645](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:02:01 dist-test-slave-7v10 krb5kdc[16645](info): set up 2 sockets
Sep 26 01:02:01 dist-test-slave-7v10 krb5kdc[16645](info): commencing operation
krb5kdc: starting...
W20260926 01:02:02.657806 3372 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Sep 26 01:02:02 dist-test-slave-7v10 krb5kdc[16645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:02:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:02:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.75.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab.
Entry for principal kudu/127.3.75.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab.
I20260926 01:02:02.773774 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:36547
--webserver_interface=127.3.75.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:43607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.75.62:36547,127.3.75.61:43445,127.3.75.60:42689
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:02:02.852783 16659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:02:02.853019 16659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:02:02.853056 16659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:02:02.854148 16659 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:02:02.854214 16659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:02:02.854235 16659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:02:02.854255 16659 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:02:02.854283 16659 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:02:02.855724 16659 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:43607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.75.62:36547,127.3.75.61:43445,127.3.75.60:42689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:36547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:02:02.855984 16659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:02:02.856206 16659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:02:02.860993 16667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:02:02.861038 16665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:02:02.861210 16659 server_base.cc:1034] running on GCE node
W20260926 01:02:02.860991 16664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:02:02.861497 16659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:02:02.861783 16659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:02:02.862920 16659 hybrid_clock.cc:648] HybridClock initialized: now 1790384522862903 us; error 33 us; skew 500 ppm
Sep 26 01:02:02 dist-test-slave-7v10 krb5kdc[16645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384522, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:02:02.869931 16659 init.cc:377] Logged in from keytab as kudu/127.3.75.62@KRBTEST.COM (short username kudu)
I20260926 01:02:02.870446 16659 webserver.cc:460] Webserver started at http://127.3.75.62:40765/ using document root <none> and password file <none>
I20260926 01:02:02.870697 16659 fs_manager.cc:359] Metadata directory not provided
I20260926 01:02:02.870765 16659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:02:02.870919 16659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:02:02.871945 16659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/minicluster-data/master-0/data/instance:
uuid: "471d9cef009a492499c7529e853dadbb"
format_stamp: "Formatted at 2026-09-26 01:02:02 on dist-test-slave-7v10"
server_key: "45f1d4aeec536090917e94574b36cbe6"
server_key_iv: "b2e1b5941899d8d27fa4f3e46d586374"
server_key_version: "encryptionkey@0"
I20260926 01:02:02.872335 16659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/minicluster-data/master-0/wal/instance:
uuid: "471d9cef009a492499c7529e853dadbb"
format_stamp: "Formatted at 2026-09-26 01:02:02 on dist-test-slave-7v10"
server_key: "45f1d4aeec536090917e94574b36cbe6"
server_key_iv: "b2e1b5941899d8d27fa4f3e46d586374"
server_key_version: "encryptionkey@0"
I20260926 01:02:02.873620 16659 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 01:02:02.874327 16674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:02:02.874498 16659 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 01:02:02.874593 16659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/minicluster-data/master-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/minicluster-data/master-0/wal
uuid: "471d9cef009a492499c7529e853dadbb"
format_stamp: "Formatted at 2026-09-26 01:02:02 on dist-test-slave-7v10"
server_key: "45f1d4aeec536090917e94574b36cbe6"
server_key_iv: "b2e1b5941899d8d27fa4f3e46d586374"
server_key_version: "encryptionkey@0"
I20260926 01:02:02.874671 16659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:02:02.885625 16659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:02:02.887341 16659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:02:02.887492 16659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:02:02.891314 16659 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.62:36547
I20260926 01:02:02.891342 16727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.62:36547 every 8 connection(s)
I20260926 01:02:02.891667 16659 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
I20260926 01:02:02.892534 16728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:02:02.894018 16728 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.75.62" port: 36547 } has no permanent_uuid. Determining permanent_uuid...
I20260926 01:02:02.898244 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 16659
I20260926 01:02:02.898365 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/minicluster-data/master-0/wal/instance
I20260926 01:02:02.898581 3372 external_mini_cluster.cc:1428] Setting key 6fdbfe84c6794ababb54be7d611ce1cc
Sep 26 01:02:02 dist-test-slave-7v10 krb5kdc[16645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384522, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.62@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 01:02:02.903246 16731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:02.894715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55138 (local address 127.3.75.62:36547)
0926 01:02:02.894857 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:02.894864 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:02.895000 (+ 136us) server_negotiation.cc:407] Connection header received
0926 01:02:02.895471 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:02.895473 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:02.895489 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:02.895567 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:02:02.896025 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:02.896795 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:02.897597 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:02.897935 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:02.898015 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:02.900209 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:02:02.900217 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:02:02.900217 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:02:02.900225 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:02:02.901725 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:02.902164 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:02.902166 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:02.902166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:02.902238 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:02.902502 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:02.902503 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:02.902503 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:02.902686 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:02:02.902748 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:02.902997 (+ 249us) server_negotiation.cc:299] Negotiation successful
0926 01:02:02.903103 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":103,"thread_start_us":72,"threads_started":1}
I20260926 01:02:02.903358 16730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:02.894681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:36547 (local address 127.0.0.1:55138)
0926 01:02:02.894844 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:02.894905 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 01:02:02.895336 (+ 431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:02:02.895578 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:02:02.895580 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:02:02.895804 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:02:02.895951 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:02.895954 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:02.896870 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:02.896870 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:02.897529 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:02.897530 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:02.898011 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:02.898012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:02.898066 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:02.898577 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:02:02.898587 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:02:02.900126 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:02:02.901805 (+ 1679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:02:02.901812 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:02:02.901814 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:02:02.902088 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:02:02.902301 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:02:02.902302 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:02:02.902302 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:02:02.902432 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:02:02.902753 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:02:02.902759 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:02:02.902837 (+ 78us) client_negotiation.cc:769] Sending connection context
0926 01:02:02.902975 (+ 138us) client_negotiation.cc:240] Negotiation successful
0926 01:02:02.903173 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":46,"thread_start_us":89,"threads_started":1}
I20260926 01:02:02.904047 16728 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.75.61" port: 43445 } has no permanent_uuid. Determining permanent_uuid...
I20260926 01:02:02.904402 16730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:02.904267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.61:43445 (local address 127.0.0.1:49360)
0926 01:02:02.904314 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:02.904371 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.61:43445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 01:02:02.904549 16678 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.75.61:43445: connect: Connection refused (error 111)
I20260926 01:02:02.904973 16730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:02.904868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.61:43445 (local address 127.0.0.1:49366)
0926 01:02:02.904914 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:02.904946 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.61:43445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 01:02:02.905085 16728 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.75.61:43445: Network error: Client connection negotiation failed: client connection to 127.3.75.61:43445: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.3.75.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.61.keytab.
Entry for principal kudu/127.3.75.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.61.keytab.
I20260926 01:02:02.922077 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.61:43445
--webserver_interface=127.3.75.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.61.keytab
--principal=kudu/127.3.75.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:43607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.75.62:36547,127.3.75.61:43445,127.3.75.60:42689
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 01:02:02.954304 16728 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.75.61" port: 43445 } attempt: 1
I20260926 01:02:02.954825 16730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:02.954666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.61:43445 (local address 127.0.0.1:49374)
0926 01:02:02.954723 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:02.954786 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.61:43445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260926 01:02:02.955237 16730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:02.955114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.61:43445 (local address 127.0.0.1:49376)
0926 01:02:02.955174 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:02.955205 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.61:43445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 01:02:02.955432 16728 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.75.61:43445: Network error: Client connection negotiation failed: client connection to 127.3.75.61:43445: connect: Connection refused (error 111)
W20260926 01:02:03.011711 16735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:02:03.011909 16735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:02:03.011937 16735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:02:03.012977 16735 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:02:03.013012 16735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:02:03.013026 16735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:02:03.013042 16735 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:02:03.013054 16735 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:02:03.014555 16735 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:43607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.75.62:36547,127.3.75.61:43445,127.3.75.60:42689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.61.keytab
--principal=kudu/127.3.75.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.61:43445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.75.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:02:03.014744 16735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:02:03.014958 16735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:02:03.019500 16743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:02:03.019495 16740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:02:03.019582 16741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:02:03.019611 16735 server_base.cc:1034] running on GCE node
I20260926 01:02:03.019909 16735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:02:03.020149 16735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:02:03.021319 16735 hybrid_clock.cc:648] HybridClock initialized: now 1790384523021302 us; error 34 us; skew 500 ppm
Sep 26 01:02:03 dist-test-slave-7v10 krb5kdc[16645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384523, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:02:03.023592 16728 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.75.61" port: 43445 } attempt: 2
I20260926 01:02:03.023988 16730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:03.023871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.61:43445 (local address 127.0.0.1:49378)
0926 01:02:03.023929 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:03.023960 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.61:43445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 01:02:03.024106 16735 init.cc:377] Logged in from keytab as kudu/127.3.75.61@KRBTEST.COM (short username kudu)
I20260926 01:02:03.024349 16730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:03.024248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.61:43445 (local address 127.0.0.1:49386)
0926 01:02:03.024294 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:03.024325 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.61:43445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 01:02:03.024435 16728 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.75.61:43445: Network error: Client connection negotiation failed: client connection to 127.3.75.61:43445: connect: Connection refused (error 111)
I20260926 01:02:03.024617 16735 webserver.cc:460] Webserver started at http://127.3.75.61:43061/ using document root <none> and password file <none>
I20260926 01:02:03.024858 16735 fs_manager.cc:359] Metadata directory not provided
I20260926 01:02:03.024921 16735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:02:03.025053 16735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:02:03.026072 16735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/minicluster-data/master-1/data/instance:
uuid: "78895aa7bd36435b8a570657fd5012b9"
format_stamp: "Formatted at 2026-09-26 01:02:03 on dist-test-slave-7v10"
server_key: "fcb0b2d7c78b0e06d8c035bb95ddeabb"
server_key_iv: "6015bc94b4eb480f9e154172c421f5be"
server_key_version: "encryptionkey@0"
I20260926 01:02:03.026427 16735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/minicluster-data/master-1/wal/instance:
uuid: "78895aa7bd36435b8a570657fd5012b9"
format_stamp: "Formatted at 2026-09-26 01:02:03 on dist-test-slave-7v10"
server_key: "fcb0b2d7c78b0e06d8c035bb95ddeabb"
server_key_iv: "6015bc94b4eb480f9e154172c421f5be"
server_key_version: "encryptionkey@0"
I20260926 01:02:03.027765 16735 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 01:02:03.028465 16750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:02:03.028615 16735 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 01:02:03.028707 16735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/minicluster-data/master-1/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/minicluster-data/master-1/wal
uuid: "78895aa7bd36435b8a570657fd5012b9"
format_stamp: "Formatted at 2026-09-26 01:02:03 on dist-test-slave-7v10"
server_key: "fcb0b2d7c78b0e06d8c035bb95ddeabb"
server_key_iv: "6015bc94b4eb480f9e154172c421f5be"
server_key_version: "encryptionkey@0"
I20260926 01:02:03.028784 16735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:02:03.038224 16735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:02:03.040480 16735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:02:03.040640 16735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:02:03.044365 16735 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.61:43445
I20260926 01:02:03.044410 16803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.61:43445 every 8 connection(s)
I20260926 01:02:03.044744 16735 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/minicluster-data/master-1/data/info.pb
I20260926 01:02:03.045575 16804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:02:03.047021 16804 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.75.62" port: 36547 } has no permanent_uuid. Determining permanent_uuid...
I20260926 01:02:03.047361 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 16735
I20260926 01:02:03.047478 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/minicluster-data/master-1/wal/instance
I20260926 01:02:03.047684 3372 external_mini_cluster.cc:1428] Setting key d69a98fdeda1242cf2ea1f91bff7c091
Sep 26 01:02:03 dist-test-slave-7v10 krb5kdc[16645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384523, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.61@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.75.60@KRBTEST.COM; defaulting to no policy
I20260926 01:02:03.057060 16731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:03.047772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55144 (local address 127.3.75.62:36547)
0926 01:02:03.047824 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:03.047827 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:03.048072 (+ 245us) server_negotiation.cc:407] Connection header received
0926 01:02:03.048770 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:03.048772 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:03.048784 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:03.048849 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:02:03.049584 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:03.050338 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:03.051424 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:03.051770 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:03.051867 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:03.054116 (+ 2249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:02:03.054119 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:02:03.054119 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:02:03.054128 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:02:03.055836 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:03.056221 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:03.056223 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:03.056223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:03.056273 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:03.056481 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:03.056481 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:03.056482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:03.056579 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:02:03.056627 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:03.056922 (+ 295us) server_negotiation.cc:299] Negotiation successful
0926 01:02:03.056992 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 01:02:03.057276 16806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:03.047771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:36547 (local address 127.0.0.1:55144)
0926 01:02:03.047944 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:03.047987 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 01:02:03.048668 (+ 681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:02:03.048934 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:02:03.048939 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:02:03.049304 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:02:03.049498 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:03.049502 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:03.050435 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:03.050435 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:03.051333 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:03.051335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:03.051872 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:03.051873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:03.051974 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:03.052484 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:02:03.052491 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:02:03.054022 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:02:03.055917 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:02:03.055929 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:02:03.055930 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:02:03.056154 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:02:03.056330 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:02:03.056331 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:02:03.056331 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:02:03.056431 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:02:03.056630 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:02:03.056634 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:02:03.056786 (+ 152us) client_negotiation.cc:769] Sending connection context
0926 01:02:03.056905 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 01:02:03.057028 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":88,"threads_started":1}
I20260926 01:02:03.057902 16804 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.75.61" port: 43445 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.60@KRBTEST.COM" created.
Sep 26 01:02:03 dist-test-slave-7v10 krb5kdc[16645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384523, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.61@KRBTEST.COM for kudu/127.3.75.61@KRBTEST.COM
I20260926 01:02:03.066119 16806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:03.058171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.61:43445 (local address 127.0.0.1:49400)
0926 01:02:03.058368 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:03.058378 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:02:03.058421 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:02:03.058571 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:02:03.058572 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:02:03.058736 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:02:03.058799 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:03.058800 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:03.060161 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:03.060162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:03.061020 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:03.061023 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:03.061464 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:03.061465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:03.061556 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:03.061906 (+ 350us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:02:03.061911 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:02:03.063215 (+ 1304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:02:03.065174 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:02:03.065175 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:02:03.065175 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:02:03.065336 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:02:03.065558 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:02:03.065559 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:02:03.065559 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:02:03.065606 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:02:03.065917 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:02:03.065917 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:02:03.065949 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 01:02:03.065993 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 01:02:03.066057 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183}
I20260926 01:02:03.066262 16808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:03.058220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49400 (local address 127.3.75.61:43445)
0926 01:02:03.058403 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:03.058408 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:03.058430 (+ 22us) server_negotiation.cc:407] Connection header received
0926 01:02:03.058463 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:03.058464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:03.058475 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:03.058561 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:02:03.058916 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:03.060073 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:03.061103 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:03.061391 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:03.061444 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:03.063295 (+ 1851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:02:03.063304 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:02:03.063304 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:02:03.063313 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:02:03.065095 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:03.065426 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:03.065428 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:03.065429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:03.065489 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:03.065659 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:03.065660 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:03.065661 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:03.065848 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:02:03.065927 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:03.066032 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 01:02:03.066114 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":151,"thread_start_us":98,"threads_started":1}
I20260926 01:02:03.066922 16804 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.75.60" port: 42689 } has no permanent_uuid. Determining permanent_uuid...
I20260926 01:02:03.067270 16806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:03.067149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.60:42689 (local address 127.0.0.1:52892)
0926 01:02:03.067196 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:03.067247 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.60:42689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 01:02:03.067411 16751 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.75.60:42689: connect: Connection refused (error 111)
I20260926 01:02:03.067807 16806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:03.067715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.60:42689 (local address 127.0.0.1:52896)
0926 01:02:03.067757 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:03.067783 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.60:42689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 01:02:03.067914 16804 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.75.60:42689: Network error: Client connection negotiation failed: client connection to 127.3.75.60:42689: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.60.keytab.
Entry for principal kudu/127.3.75.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.60.keytab.
I20260926 01:02:03.073836 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.60:42689
--webserver_interface=127.3.75.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.60.keytab
--principal=kudu/127.3.75.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:43607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.75.62:36547,127.3.75.61:43445,127.3.75.60:42689
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 01:02:03.115599 16728 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.75.61" port: 43445 } attempt: 3
I20260926 01:02:03.117136 16804 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.75.60" port: 42689 } attempt: 1
I20260926 01:02:03.117477 16806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:03.117379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.60:42689 (local address 127.0.0.1:52904)
0926 01:02:03.117427 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:03.117451 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.60:42689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 01:02:03.117798 16806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:03.117700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.60:42689 (local address 127.0.0.1:52920)
0926 01:02:03.117748 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:03.117777 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.60:42689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 01:02:03.117913 16804 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.75.60:42689: Network error: Client connection negotiation failed: client connection to 127.3.75.60:42689: connect: Connection refused (error 111)
Sep 26 01:02:03 dist-test-slave-7v10 krb5kdc[16645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384522, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.62@KRBTEST.COM for kudu/127.3.75.61@KRBTEST.COM
I20260926 01:02:03.123016 16808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:03.116162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49416 (local address 127.3.75.61:43445)
0926 01:02:03.116229 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:03.116232 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:03.116243 (+ 11us) server_negotiation.cc:407] Connection header received
0926 01:02:03.116282 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:03.116283 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:03.116294 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:03.116374 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:02:03.116725 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:03.117305 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:03.118270 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:03.118471 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:03.118528 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:03.120515 (+ 1987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:02:03.120522 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:02:03.120522 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:02:03.120533 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:02:03.121904 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:03.122249 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:03.122250 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:03.122250 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:03.122308 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:03.122500 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:03.122501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:03.122501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:03.122620 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:02:03.122691 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:03.122839 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 01:02:03.122930 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 01:02:03.123021 16730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:03.116070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.61:43445 (local address 127.0.0.1:49416)
0926 01:02:03.116150 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:03.116164 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 01:02:03.116234 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:02:03.116398 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:02:03.116399 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:02:03.116570 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:02:03.116638 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:03.116640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:03.117406 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:03.117407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:03.118188 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:03.118191 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:03.118549 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:03.118550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:03.118651 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:03.119132 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:02:03.119138 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:02:03.120429 (+ 1291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:02:03.122006 (+ 1577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:02:03.122008 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:02:03.122009 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:02:03.122181 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:02:03.122375 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:02:03.122376 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:02:03.122376 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:02:03.122440 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:02:03.122682 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:02:03.122683 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:02:03.122729 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 01:02:03.122809 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 01:02:03.122930 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 01:02:03.123390 16728 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.75.60" port: 42689 } has no permanent_uuid. Determining permanent_uuid...
I20260926 01:02:03.123719 16730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:03.123630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.60:42689 (local address 127.0.0.1:52930)
0926 01:02:03.123673 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:03.123701 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.60:42689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260926 01:02:03.124058 16730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:03.123974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.60:42689 (local address 127.0.0.1:52936)
0926 01:02:03.124014 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:03.124043 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.60:42689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 01:02:03.124143 16728 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.75.60:42689: Network error: Client connection negotiation failed: client connection to 127.3.75.60:42689: connect: Connection refused (error 111)
I20260926 01:02:03.155277 16728 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.75.60" port: 42689 } attempt: 1
I20260926 01:02:03.155658 16730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:03.155542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.60:42689 (local address 127.0.0.1:52948)
0926 01:02:03.155600 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:03.155632 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.60:42689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 01:02:03.155642 16811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:02:03.155897 16811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:02:03.155957 16811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 01:02:03.156010 16730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:03.155892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.60:42689 (local address 127.0.0.1:52954)
0926 01:02:03.155947 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:03.155987 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.60:42689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 01:02:03.156111 16728 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.75.60:42689: Network error: Client connection negotiation failed: client connection to 127.3.75.60:42689: connect: Connection refused (error 111)
W20260926 01:02:03.157191 16811 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:02:03.157233 16811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:02:03.157248 16811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:02:03.157284 16811 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:02:03.157310 16811 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:02:03.158818 16811 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:43607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.75.62:36547,127.3.75.61:43445,127.3.75.60:42689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.60.keytab
--principal=kudu/127.3.75.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.60:42689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.75.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:02:03.159086 16811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:02:03.159303 16811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:02:03.164062 16816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:02:03.164047 16819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:02:03.164209 16811 server_base.cc:1034] running on GCE node
W20260926 01:02:03.164047 16817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:02:03.164533 16811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:02:03.164780 16811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:02:03.165952 16811 hybrid_clock.cc:648] HybridClock initialized: now 1790384523165933 us; error 40 us; skew 500 ppm
Sep 26 01:02:03 dist-test-slave-7v10 krb5kdc[16645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384523, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:02:03.168355 16811 init.cc:377] Logged in from keytab as kudu/127.3.75.60@KRBTEST.COM (short username kudu)
I20260926 01:02:03.168742 16811 webserver.cc:460] Webserver started at http://127.3.75.60:33909/ using document root <none> and password file <none>
I20260926 01:02:03.168929 16811 fs_manager.cc:359] Metadata directory not provided
I20260926 01:02:03.168969 16811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:02:03.169060 16811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:02:03.170073 16811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/minicluster-data/master-2/data/instance:
uuid: "7faf29afd8094c79b10aa1d9aacf8de2"
format_stamp: "Formatted at 2026-09-26 01:02:03 on dist-test-slave-7v10"
server_key: "ae4df4a5423acb87e2f1166e29a68d40"
server_key_iv: "99c075e674c08f00346ea3f1261bbf2f"
server_key_version: "encryptionkey@0"
I20260926 01:02:03.170362 16811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/minicluster-data/master-2/wal/instance:
uuid: "7faf29afd8094c79b10aa1d9aacf8de2"
format_stamp: "Formatted at 2026-09-26 01:02:03 on dist-test-slave-7v10"
server_key: "ae4df4a5423acb87e2f1166e29a68d40"
server_key_iv: "99c075e674c08f00346ea3f1261bbf2f"
server_key_version: "encryptionkey@0"
I20260926 01:02:03.171722 16811 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:02:03.172392 16826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:02:03.172547 16811 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:02:03.172627 16811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/minicluster-data/master-2/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/minicluster-data/master-2/wal
uuid: "7faf29afd8094c79b10aa1d9aacf8de2"
format_stamp: "Formatted at 2026-09-26 01:02:03 on dist-test-slave-7v10"
server_key: "ae4df4a5423acb87e2f1166e29a68d40"
server_key_iv: "99c075e674c08f00346ea3f1261bbf2f"
server_key_version: "encryptionkey@0"
I20260926 01:02:03.172678 16811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:02:03.181526 16811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:02:03.183256 16811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:02:03.183363 16811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:02:03.186056 16804 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.75.60" port: 42689 } attempt: 2
I20260926 01:02:03.186563 16806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:03.186392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.60:42689 (local address 127.0.0.1:52956)
0926 01:02:03.186468 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:03.186512 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.60:42689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 01:02:03.186975 16806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:03.186837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.60:42689 (local address 127.0.0.1:52970)
0926 01:02:03.186910 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:03.186941 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.60:42689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 01:02:03.187088 16804 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.75.60:42689: Network error: Client connection negotiation failed: client connection to 127.3.75.60:42689: connect: Connection refused (error 111)
I20260926 01:02:03.187297 16811 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.60:42689
I20260926 01:02:03.187361 16879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.60:42689 every 8 connection(s)
I20260926 01:02:03.187661 16811 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/minicluster-data/master-2/data/info.pb
I20260926 01:02:03.188102 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 16811
I20260926 01:02:03.188252 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384372073861-3372-0/minicluster-data/master-2/wal/instance
I20260926 01:02:03.188459 3372 external_mini_cluster.cc:1428] Setting key 8467de8f6810e1adc8db3c44038ca76a
I20260926 01:02:03.188719 16880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:02:03.190641 16880 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.75.62" port: 36547 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 01:02:03 dist-test-slave-7v10 krb5kdc[16645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.75.60@KRBTEST.COM
Sep 26 01:02:03 dist-test-slave-7v10 krb5kdc[16645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384523, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.60@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 01:02:03.198559 16731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:03.191296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55146 (local address 127.3.75.62:36547)
0926 01:02:03.191345 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:03.191347 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:03.191503 (+ 156us) server_negotiation.cc:407] Connection header received
0926 01:02:03.191578 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:03.191579 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:03.191590 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:03.191647 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:02:03.192069 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:03.192535 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:03.193280 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:03.193491 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:03.193563 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:03.196008 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:02:03.196009 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:02:03.196009 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:02:03.196016 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:02:03.197316 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:03.197672 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:03.197673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:03.197674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:03.197730 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:03.197955 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:03.197956 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:03.197956 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:03.198060 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:02:03.198125 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:03.198362 (+ 237us) server_negotiation.cc:299] Negotiation successful
0926 01:02:03.198443 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 01:02:03.198727 16884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:03.191237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:36547 (local address 127.0.0.1:55146)
0926 01:02:03.191382 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:03.191436 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 01:02:03.191522 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:02:03.191664 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:02:03.191667 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:02:03.191924 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:02:03.191987 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:03.191989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:03.192618 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:03.192619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:03.193195 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:03.193196 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:03.193569 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:03.193571 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:03.193646 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:03.194074 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:02:03.194081 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:02:03.195925 (+ 1844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:02:03.197403 (+ 1478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:02:03.197410 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:02:03.197412 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:02:03.197601 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:02:03.197792 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:02:03.197793 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:02:03.197794 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:02:03.197882 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:02:03.198128 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:02:03.198135 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:02:03.198204 (+ 69us) client_negotiation.cc:769] Sending connection context
0926 01:02:03.198333 (+ 129us) client_negotiation.cc:240] Negotiation successful
0926 01:02:03.198559 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":84,"thread_start_us":84,"threads_started":1}
I20260926 01:02:03.198731 16883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:03.188889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52982 (local address 127.3.75.60:42689)
0926 01:02:03.189069 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:03.189079 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:03.189110 (+ 31us) server_negotiation.cc:407] Connection header received
0926 01:02:03.189766 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:03.189772 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:03.189820 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:03.189993 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:02:03.190611 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:03.191815 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:03.192544 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:03.192959 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:03.193076 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:03.195492 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:02:03.195504 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:02:03.195504 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:02:03.195510 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:02:03.197244 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:03.197595 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:03.197598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:03.197598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:03.197676 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:03.197883 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:03.197883 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:03.197884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:03.198072 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:02:03.198203 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:03.198333 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 01:02:03.198474 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":83,"threads_started":1}
I20260926 01:02:03.199308 16880 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.75.61" port: 43445 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 01:02:03 dist-test-slave-7v10 krb5kdc[16645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384523, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.60@KRBTEST.COM for kudu/127.3.75.61@KRBTEST.COM
Sep 26 01:02:03 dist-test-slave-7v10 krb5kdc[16645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 01:02:03.205696 16808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:03.199632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49432 (local address 127.3.75.61:43445)
0926 01:02:03.199745 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:03.199747 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:03.199756 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:02:03.199779 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:03.199780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:03.199789 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:03.199852 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:02:03.200080 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:03.200533 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:03.201154 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:03.201362 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:03.201441 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:03.203188 (+ 1747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:02:03.203189 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:02:03.203189 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:02:03.203195 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:02:03.204620 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:03.204967 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:03.204968 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:03.204969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:03.205023 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:03.205210 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:03.205211 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:03.205211 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:03.205356 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:02:03.205408 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:03.205574 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 01:02:03.205633 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260926 01:02:03.205750 16884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:03.199579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.61:43445 (local address 127.0.0.1:49432)
0926 01:02:03.199666 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:03.199676 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:02:03.199714 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:02:03.199856 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:02:03.199857 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:02:03.199987 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:02:03.200013 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:03.200014 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:03.200610 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:03.200611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:03.201079 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:03.201080 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:03.201442 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:03.201443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:03.201499 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:03.201895 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:02:03.201900 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:02:03.203107 (+ 1207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:02:03.204718 (+ 1611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:02:03.204719 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:02:03.204720 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:02:03.204885 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:02:03.205088 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:02:03.205089 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:02:03.205089 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:02:03.205142 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:02:03.205421 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:02:03.205422 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:02:03.205470 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 01:02:03.205556 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 01:02:03.205644 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260926 01:02:03.206120 16880 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.75.60" port: 42689 } has no permanent_uuid. Determining permanent_uuid...
I20260926 01:02:03.207237 16731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:03.199689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55160 (local address 127.3.75.62:36547)
0926 01:02:03.199731 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:03.199733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:03.199741 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:02:03.199772 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:03.199773 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:03.199781 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:03.199821 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:02:03.200250 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:03.200681 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:03.201314 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:03.201507 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:03.201577 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:03.203839 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:02:03.203840 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:02:03.203841 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:02:03.203860 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:02:03.206090 (+ 2230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:03.206491 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:03.206493 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:03.206493 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:03.206553 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:03.206749 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:03.206749 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:03.206750 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:03.206900 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:02:03.207068 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:03.207098 (+ 30us) server_negotiation.cc:299] Negotiation successful
0926 01:02:03.207159 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Sep 26 01:02:03 dist-test-slave-7v10 krb5kdc[16645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384523, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.60@KRBTEST.COM for kudu/127.3.75.60@KRBTEST.COM
Sep 26 01:02:03 dist-test-slave-7v10 krb5kdc[16645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.75.61@KRBTEST.COM
I20260926 01:02:03.212766 16883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:03.206441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52998 (local address 127.3.75.60:42689)
0926 01:02:03.206478 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:03.206480 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:03.206524 (+ 44us) server_negotiation.cc:407] Connection header received
0926 01:02:03.206610 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:03.206611 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:03.206620 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:03.206669 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:02:03.206958 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:03.207629 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:03.208281 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:03.208562 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:03.208610 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:03.210152 (+ 1542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:02:03.210153 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:02:03.210153 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:02:03.210159 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:02:03.211661 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:03.212017 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:03.212018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:03.212018 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:03.212062 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:03.212265 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:03.212266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:03.212266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:03.212397 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:02:03.212465 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:03.212601 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 01:02:03.212686 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":19}
I20260926 01:02:03.212780 16884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:03.206393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.60:42689 (local address 127.0.0.1:52998)
0926 01:02:03.206479 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:03.206489 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:02:03.206539 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:02:03.206670 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:02:03.206671 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:02:03.206821 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:02:03.206866 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:03.206868 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:03.207687 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:03.207689 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:03.208207 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:03.208209 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:03.208625 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:03.208626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:03.208680 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:03.208992 (+ 312us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:02:03.208996 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:02:03.210075 (+ 1079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:02:03.211731 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:02:03.211732 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:02:03.211732 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:02:03.211950 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:02:03.212119 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:02:03.212120 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:02:03.212121 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:02:03.212196 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:02:03.212458 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:02:03.212459 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:02:03.212492 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 01:02:03.212545 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 01:02:03.212642 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260926 01:02:03.214677 16880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7faf29afd8094c79b10aa1d9aacf8de2: Bootstrap starting.
I20260926 01:02:03.215451 16808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:03.207825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49438 (local address 127.3.75.61:43445)
0926 01:02:03.207866 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:03.207868 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:03.207876 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:02:03.207895 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:03.207895 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:03.207903 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:03.207957 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:02:03.208438 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:03.209048 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:03.209686 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:03.209929 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:03.209987 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:03.212628 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:02:03.212629 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:02:03.212630 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:02:03.212635 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:02:03.214268 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:03.214567 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:03.214568 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:03.214569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:03.214625 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:03.214851 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:03.214852 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:03.214853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:03.214996 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:02:03.215205 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:03.215254 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 01:02:03.215344 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 01:02:03.215803 16880 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7faf29afd8094c79b10aa1d9aacf8de2: Neither blocks nor log segments found. Creating new log.
I20260926 01:02:03.216145 16880 log.cc:824] T 00000000000000000000000000000000 P 7faf29afd8094c79b10aa1d9aacf8de2: Log is configured to *not* fsync() on all Append() calls
I20260926 01:02:03.217012 16880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7faf29afd8094c79b10aa1d9aacf8de2: No bootstrap required, opened a new log
I20260926 01:02:03.217005 3372 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:02:03.218592 16880 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7faf29afd8094c79b10aa1d9aacf8de2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "471d9cef009a492499c7529e853dadbb" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 36547 } } peers { permanent_uuid: "78895aa7bd36435b8a570657fd5012b9" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 43445 } } peers { permanent_uuid: "7faf29afd8094c79b10aa1d9aacf8de2" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 42689 } }
I20260926 01:02:03.218780 16880 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7faf29afd8094c79b10aa1d9aacf8de2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:02:03.218832 16880 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7faf29afd8094c79b10aa1d9aacf8de2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7faf29afd8094c79b10aa1d9aacf8de2, State: Initialized, Role: FOLLOWER
I20260926 01:02:03.219020 16880 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7faf29afd8094c79b10aa1d9aacf8de2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "471d9cef009a492499c7529e853dadbb" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 36547 } } peers { permanent_uuid: "78895aa7bd36435b8a570657fd5012b9" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 43445 } } peers { permanent_uuid: "7faf29afd8094c79b10aa1d9aacf8de2" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 42689 } }
I20260926 01:02:03.219360 16880 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7faf29afd8094c79b10aa1d9aacf8de2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:02:03.220609 16893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7faf29afd8094c79b10aa1d9aacf8de2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "471d9cef009a492499c7529e853dadbb" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 36547 } } peers { permanent_uuid: "78895aa7bd36435b8a570657fd5012b9" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 43445 } } peers { permanent_uuid: "7faf29afd8094c79b10aa1d9aacf8de2" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 42689 } } }
I20260926 01:02:03.220925 16893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7faf29afd8094c79b10aa1d9aacf8de2 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 01:02:03.224119 16904 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7faf29afd8094c79b10aa1d9aacf8de2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:02:03.224186 16904 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:02:03.221678 16811 master_runner.cc:186] Error getting master registration for 127.3.75.62:36547: OK, instance_id { permanent_uuid: "471d9cef009a492499c7529e853dadbb" instance_seqno: 1790384522890163 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 01:02:03.224833 16811 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 01:02:03.226511 16731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:03.218483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55174 (local address 127.3.75.62:36547)
0926 01:02:03.218551 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:03.218553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:03.218630 (+ 77us) server_negotiation.cc:407] Connection header received
0926 01:02:03.218732 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:03.218733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:03.218743 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:03.218777 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:02:03.219600 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:03.220334 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:03.221250 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:03.221551 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:03.221615 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:03.223204 (+ 1589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:02:03.223206 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:02:03.223206 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:02:03.223214 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:02:03.224994 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:03.225478 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:03.225479 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:03.225479 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:03.225540 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:03.225738 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:03.225739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:03.225739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:03.225871 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:02:03.225961 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:03.226326 (+ 365us) server_negotiation.cc:299] Negotiation successful
0926 01:02:03.226408 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 01:02:03.226631 16808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:03.218441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49446 (local address 127.3.75.61:43445)
0926 01:02:03.218522 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:03.218523 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:03.218838 (+ 315us) server_negotiation.cc:407] Connection header received
0926 01:02:03.219083 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:03.219084 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:03.219094 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:03.219123 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:02:03.219981 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:03.220630 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:03.221781 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:03.222093 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:03.222139 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:03.223614 (+ 1475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:02:03.223615 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:02:03.223615 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:02:03.223621 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:02:03.225251 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:03.225658 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:03.225659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:03.225659 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:03.225718 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:03.225963 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:03.225964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:03.225964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:03.226107 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:02:03.226195 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:03.226427 (+ 232us) server_negotiation.cc:299] Negotiation successful
0926 01:02:03.226519 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 01:02:03.228201 16883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:03.218463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53006 (local address 127.3.75.60:42689)
0926 01:02:03.218498 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:03.218500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:03.218667 (+ 167us) server_negotiation.cc:407] Connection header received
0926 01:02:03.218715 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:03.218716 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:03.218726 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:03.218786 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:02:03.219586 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:03.220319 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:03.222363 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:03.222961 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:03.223006 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:03.225392 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:02:03.225394 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:02:03.225394 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:02:03.225402 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:02:03.227110 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:03.227492 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:03.227493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:03.227494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:03.227554 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:03.227698 (+ 144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:03.227698 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:03.227698 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:03.227857 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:02:03.227949 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:03.228042 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 01:02:03.228117 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 01:02:03.231248 16728 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.75.60" port: 42689 } attempt: 2
Sep 26 01:02:03 dist-test-slave-7v10 krb5kdc[16645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384522, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.62@KRBTEST.COM for kudu/127.3.75.60@KRBTEST.COM
I20260926 01:02:03.237850 16730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:03.231493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.60:42689 (local address 127.0.0.1:53010)
0926 01:02:03.231540 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:03.231551 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:02:03.231589 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:02:03.231716 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:02:03.231717 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:02:03.231833 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:02:03.231878 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:03.231879 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:03.232769 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:03.232769 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:03.233262 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:03.233264 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:03.233658 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:03.233659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:03.233692 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:03.234133 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:02:03.234138 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:02:03.235438 (+ 1300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:02:03.237022 (+ 1584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:02:03.237023 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:02:03.237023 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:02:03.237185 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:02:03.237370 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:02:03.237370 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:02:03.237371 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:02:03.237417 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:02:03.237651 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:02:03.237652 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:02:03.237683 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 01:02:03.237729 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 01:02:03.237790 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 01:02:03.237918 16883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:03.231548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53010 (local address 127.3.75.60:42689)
0926 01:02:03.231582 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:03.231584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:03.231592 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:02:03.231641 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:03.231642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:03.231652 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:03.231713 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:02:03.231942 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:03.232701 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:03.233331 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:03.233591 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:03.233659 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:03.235519 (+ 1860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:02:03.235520 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:02:03.235521 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:02:03.235527 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:02:03.236950 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:03.237249 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:03.237250 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:03.237250 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:03.237305 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:03.237470 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:03.237471 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:03.237471 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:03.237593 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:02:03.237655 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:03.237759 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 01:02:03.237834 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 01:02:03.239476 16728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 471d9cef009a492499c7529e853dadbb: Bootstrap starting.
I20260926 01:02:03.240199 16728 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 471d9cef009a492499c7529e853dadbb: Neither blocks nor log segments found. Creating new log.
I20260926 01:02:03.240481 16728 log.cc:824] T 00000000000000000000000000000000 P 471d9cef009a492499c7529e853dadbb: Log is configured to *not* fsync() on all Append() calls
I20260926 01:02:03.241262 16728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 471d9cef009a492499c7529e853dadbb: No bootstrap required, opened a new log
I20260926 01:02:03.243256 16728 raft_consensus.cc:348] T 00000000000000000000000000000000 P 471d9cef009a492499c7529e853dadbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "471d9cef009a492499c7529e853dadbb" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 36547 } } peers { permanent_uuid: "78895aa7bd36435b8a570657fd5012b9" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 43445 } } peers { permanent_uuid: "7faf29afd8094c79b10aa1d9aacf8de2" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 42689 } }
I20260926 01:02:03.243391 16728 raft_consensus.cc:374] T 00000000000000000000000000000000 P 471d9cef009a492499c7529e853dadbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:02:03.243427 16728 raft_consensus.cc:729] T 00000000000000000000000000000000 P 471d9cef009a492499c7529e853dadbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 471d9cef009a492499c7529e853dadbb, State: Initialized, Role: FOLLOWER
I20260926 01:02:03.243546 16728 consensus_queue.cc:260] T 00000000000000000000000000000000 P 471d9cef009a492499c7529e853dadbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "471d9cef009a492499c7529e853dadbb" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 36547 } } peers { permanent_uuid: "78895aa7bd36435b8a570657fd5012b9" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 43445 } } peers { permanent_uuid: "7faf29afd8094c79b10aa1d9aacf8de2" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 42689 } }
I20260926 01:02:03.243842 16728 sys_catalog.cc:564] T 00000000000000000000000000000000 P 471d9cef009a492499c7529e853dadbb [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:02:03.243808 16907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 471d9cef009a492499c7529e853dadbb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "471d9cef009a492499c7529e853dadbb" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 36547 } } peers { permanent_uuid: "78895aa7bd36435b8a570657fd5012b9" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 43445 } } peers { permanent_uuid: "7faf29afd8094c79b10aa1d9aacf8de2" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 42689 } } }
I20260926 01:02:03.243921 16907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 471d9cef009a492499c7529e853dadbb [sys.catalog]: This master's current role is: FOLLOWER
W20260926 01:02:03.245627 16918 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 471d9cef009a492499c7529e853dadbb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:02:03.245719 16918 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:02:03.246234 16659 master_runner.cc:186] Error getting master registration for 127.3.75.61:43445: OK, instance_id { permanent_uuid: "78895aa7bd36435b8a570657fd5012b9" instance_seqno: 1790384523043332 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 01:02:03.248569 16659 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 01:02:03.278229 16804 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.75.60" port: 42689 } attempt: 3
Sep 26 01:02:03 dist-test-slave-7v10 krb5kdc[16645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384523, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.61@KRBTEST.COM for kudu/127.3.75.60@KRBTEST.COM
I20260926 01:02:03.285125 16806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:03.278571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.60:42689 (local address 127.0.0.1:53020)
0926 01:02:03.278622 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:03.278633 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:02:03.278677 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:02:03.278823 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:02:03.278824 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:02:03.279016 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:02:03.279058 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:03.279060 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:03.279696 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:03.279697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:03.280209 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:03.280210 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:03.280578 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:03.280579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:03.280644 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:03.281096 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:02:03.281106 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:02:03.282265 (+ 1159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:02:03.283825 (+ 1560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:02:03.283826 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:02:03.283827 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:02:03.284020 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:02:03.284222 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:02:03.284223 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:02:03.284223 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:02:03.284285 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:02:03.284786 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:02:03.284787 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:02:03.284833 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 01:02:03.284902 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 01:02:03.285009 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 01:02:03.285493 16883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:03.278620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53020 (local address 127.3.75.60:42689)
0926 01:02:03.278660 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:03.278662 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:03.278672 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:02:03.278738 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:03.278739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:03.278749 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:03.278818 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:02:03.279135 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:03.279610 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:03.280287 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:03.280491 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:03.280580 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:03.282351 (+ 1771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:02:03.282353 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:02:03.282353 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:02:03.282364 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:02:03.283729 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:03.284090 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:03.284091 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:03.284092 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:03.284156 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:03.284544 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:03.284545 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:03.284545 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:03.284679 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:02:03.284738 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:03.285315 (+ 577us) server_negotiation.cc:299] Negotiation successful
0926 01:02:03.285399 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 01:02:03.286744 16804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78895aa7bd36435b8a570657fd5012b9: Bootstrap starting.
I20260926 01:02:03.287395 16804 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78895aa7bd36435b8a570657fd5012b9: Neither blocks nor log segments found. Creating new log.
I20260926 01:02:03.287696 16804 log.cc:824] T 00000000000000000000000000000000 P 78895aa7bd36435b8a570657fd5012b9: Log is configured to *not* fsync() on all Append() calls
I20260926 01:02:03.288479 16804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78895aa7bd36435b8a570657fd5012b9: No bootstrap required, opened a new log
I20260926 01:02:03.289907 16804 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78895aa7bd36435b8a570657fd5012b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "471d9cef009a492499c7529e853dadbb" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 36547 } } peers { permanent_uuid: "78895aa7bd36435b8a570657fd5012b9" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 43445 } } peers { permanent_uuid: "7faf29afd8094c79b10aa1d9aacf8de2" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 42689 } }
I20260926 01:02:03.290043 16804 raft_consensus.cc:374] T 00000000000000000000000000000000 P 78895aa7bd36435b8a570657fd5012b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:02:03.290086 16804 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78895aa7bd36435b8a570657fd5012b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78895aa7bd36435b8a570657fd5012b9, State: Initialized, Role: FOLLOWER
I20260926 01:02:03.290212 16804 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78895aa7bd36435b8a570657fd5012b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "471d9cef009a492499c7529e853dadbb" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 36547 } } peers { permanent_uuid: "78895aa7bd36435b8a570657fd5012b9" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 43445 } } peers { permanent_uuid: "7faf29afd8094c79b10aa1d9aacf8de2" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 42689 } }
I20260926 01:02:03.290485 16921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78895aa7bd36435b8a570657fd5012b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "471d9cef009a492499c7529e853dadbb" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 36547 } } peers { permanent_uuid: "78895aa7bd36435b8a570657fd5012b9" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 43445 } } peers { permanent_uuid: "7faf29afd8094c79b10aa1d9aacf8de2" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 42689 } } }
I20260926 01:02:03.290546 16804 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78895aa7bd36435b8a570657fd5012b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:02:03.290644 16921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78895aa7bd36435b8a570657fd5012b9 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 01:02:03.291188 16756 tablet.cc:2369] T 00000000000000000000000000000000 P 78895aa7bd36435b8a570657fd5012b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 01:02:03.292760 16932 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 78895aa7bd36435b8a570657fd5012b9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:02:03.292886 16932 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:02:03.295730 16735 master_runner.cc:263] No leader master found from master 78895aa7bd36435b8a570657fd5012b9
I20260926 01:02:03.295778 16735 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 01:02:03.334685 16907 raft_consensus.cc:482] T 00000000000000000000000000000000 P 471d9cef009a492499c7529e853dadbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:02:03.334822 16907 raft_consensus.cc:504] T 00000000000000000000000000000000 P 471d9cef009a492499c7529e853dadbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "471d9cef009a492499c7529e853dadbb" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 36547 } } peers { permanent_uuid: "78895aa7bd36435b8a570657fd5012b9" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 43445 } } peers { permanent_uuid: "7faf29afd8094c79b10aa1d9aacf8de2" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 42689 } }
I20260926 01:02:03.335083 16907 leader_election.cc:288] T 00000000000000000000000000000000 P 471d9cef009a492499c7529e853dadbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78895aa7bd36435b8a570657fd5012b9 (127.3.75.61:43445), 7faf29afd8094c79b10aa1d9aacf8de2 (127.3.75.60:42689)
I20260926 01:02:03.335559 16778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "471d9cef009a492499c7529e853dadbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78895aa7bd36435b8a570657fd5012b9" is_pre_election: true
I20260926 01:02:03.335709 16778 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 78895aa7bd36435b8a570657fd5012b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 471d9cef009a492499c7529e853dadbb in term 0.
I20260926 01:02:03.336124 16854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "471d9cef009a492499c7529e853dadbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7faf29afd8094c79b10aa1d9aacf8de2" is_pre_election: true
I20260926 01:02:03.336314 16854 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7faf29afd8094c79b10aa1d9aacf8de2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 471d9cef009a492499c7529e853dadbb in term 0.
I20260926 01:02:03.336585 16678 leader_election.cc:302] T 00000000000000000000000000000000 P 471d9cef009a492499c7529e853dadbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 471d9cef009a492499c7529e853dadbb, 78895aa7bd36435b8a570657fd5012b9; no voters:
I20260926 01:02:03.336805 16907 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 471d9cef009a492499c7529e853dadbb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:02:03.336864 16907 raft_consensus.cc:482] T 00000000000000000000000000000000 P 471d9cef009a492499c7529e853dadbb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:02:03.336887 16907 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 471d9cef009a492499c7529e853dadbb [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:02:03.337720 16907 raft_consensus.cc:504] T 00000000000000000000000000000000 P 471d9cef009a492499c7529e853dadbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "471d9cef009a492499c7529e853dadbb" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 36547 } } peers { permanent_uuid: "78895aa7bd36435b8a570657fd5012b9" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 43445 } } peers { permanent_uuid: "7faf29afd8094c79b10aa1d9aacf8de2" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 42689 } }
I20260926 01:02:03.337832 16907 leader_election.cc:288] T 00000000000000000000000000000000 P 471d9cef009a492499c7529e853dadbb [CANDIDATE]: Term 1 election: Requested vote from peers 78895aa7bd36435b8a570657fd5012b9 (127.3.75.61:43445), 7faf29afd8094c79b10aa1d9aacf8de2 (127.3.75.60:42689)
I20260926 01:02:03.338099 16778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "471d9cef009a492499c7529e853dadbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78895aa7bd36435b8a570657fd5012b9"
I20260926 01:02:03.338169 16778 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78895aa7bd36435b8a570657fd5012b9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:02:03.339982 16854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "471d9cef009a492499c7529e853dadbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7faf29afd8094c79b10aa1d9aacf8de2"
I20260926 01:02:03.340055 16854 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7faf29afd8094c79b10aa1d9aacf8de2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:02:03.340870 16854 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7faf29afd8094c79b10aa1d9aacf8de2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 471d9cef009a492499c7529e853dadbb in term 1.
I20260926 01:02:03.341149 16675 leader_election.cc:302] T 00000000000000000000000000000000 P 471d9cef009a492499c7529e853dadbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 471d9cef009a492499c7529e853dadbb, 7faf29afd8094c79b10aa1d9aacf8de2; no voters:
I20260926 01:02:03.341440 16778 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 78895aa7bd36435b8a570657fd5012b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 471d9cef009a492499c7529e853dadbb in term 1.
I20260926 01:02:03.342180 16907 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 471d9cef009a492499c7529e853dadbb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:02:03.342340 16907 raft_consensus.cc:686] T 00000000000000000000000000000000 P 471d9cef009a492499c7529e853dadbb [term 1 LEADER]: Becoming Leader. State: Replica: 471d9cef009a492499c7529e853dadbb, State: Running, Role: LEADER
I20260926 01:02:03.342422 16907 consensus_queue.cc:237] T 00000000000000000000000000000000 P 471d9cef009a492499c7529e853dadbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "471d9cef009a492499c7529e853dadbb" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 36547 } } peers { permanent_uuid: "78895aa7bd36435b8a570657fd5012b9" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 43445 } } peers { permanent_uuid: "7faf29afd8094c79b10aa1d9aacf8de2" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 42689 } }
I20260926 01:02:03.342770 16907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 471d9cef009a492499c7529e853dadbb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 471d9cef009a492499c7529e853dadbb. Latest consensus state: current_term: 1 leader_uuid: "471d9cef009a492499c7529e853dadbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "471d9cef009a492499c7529e853dadbb" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 36547 } } peers { permanent_uuid: "78895aa7bd36435b8a570657fd5012b9" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 43445 } } peers { permanent_uuid: "7faf29afd8094c79b10aa1d9aacf8de2" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 42689 } } }
I20260926 01:02:03.342833 16907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 471d9cef009a492499c7529e853dadbb [sys.catalog]: This master's current role is: LEADER
I20260926 01:02:03.343168 16936 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:02:03.343394 16936 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:02:03.345431 16854 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7faf29afd8094c79b10aa1d9aacf8de2 [term 1 FOLLOWER]: Refusing update from remote peer 471d9cef009a492499c7529e853dadbb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:02:03.345652 16778 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 78895aa7bd36435b8a570657fd5012b9 [term 1 FOLLOWER]: Refusing update from remote peer 471d9cef009a492499c7529e853dadbb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:02:03.346123 16907 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 471d9cef009a492499c7529e853dadbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "78895aa7bd36435b8a570657fd5012b9" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 43445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:02:03.346252 16907 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 471d9cef009a492499c7529e853dadbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7faf29afd8094c79b10aa1d9aacf8de2" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 42689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:02:03.347496 16921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78895aa7bd36435b8a570657fd5012b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 471d9cef009a492499c7529e853dadbb. Latest consensus state: current_term: 1 leader_uuid: "471d9cef009a492499c7529e853dadbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "471d9cef009a492499c7529e853dadbb" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 36547 } } peers { permanent_uuid: "78895aa7bd36435b8a570657fd5012b9" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 43445 } } peers { permanent_uuid: "7faf29afd8094c79b10aa1d9aacf8de2" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 42689 } } }
I20260926 01:02:03.347577 16921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78895aa7bd36435b8a570657fd5012b9 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:02:03.348552 16893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7faf29afd8094c79b10aa1d9aacf8de2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 471d9cef009a492499c7529e853dadbb. Latest consensus state: current_term: 1 leader_uuid: "471d9cef009a492499c7529e853dadbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "471d9cef009a492499c7529e853dadbb" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 36547 } } peers { permanent_uuid: "78895aa7bd36435b8a570657fd5012b9" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 43445 } } peers { permanent_uuid: "7faf29afd8094c79b10aa1d9aacf8de2" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 42689 } } }
I20260926 01:02:03.348649 16893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7faf29afd8094c79b10aa1d9aacf8de2 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:02:03.350178 16907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 471d9cef009a492499c7529e853dadbb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "471d9cef009a492499c7529e853dadbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "471d9cef009a492499c7529e853dadbb" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 36547 } } peers { permanent_uuid: "78895aa7bd36435b8a570657fd5012b9" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 43445 } } peers { permanent_uuid: "7faf29afd8094c79b10aa1d9aacf8de2" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 42689 } } }
I20260926 01:02:03.350251 16907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 471d9cef009a492499c7529e853dadbb [sys.catalog]: This master's current role is: LEADER
I20260926 01:02:03.350625 16893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7faf29afd8094c79b10aa1d9aacf8de2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "471d9cef009a492499c7529e853dadbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "471d9cef009a492499c7529e853dadbb" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 36547 } } peers { permanent_uuid: "78895aa7bd36435b8a570657fd5012b9" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 43445 } } peers { permanent_uuid: "7faf29afd8094c79b10aa1d9aacf8de2" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 42689 } } }
I20260926 01:02:03.350696 16893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7faf29afd8094c79b10aa1d9aacf8de2 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:02:03.350973 16937 mvcc.cc:205] Tried to move back new op lower bound from 7333415007620259840 to 7333415007611023360. Current Snapshot: MvccSnapshot[applied={T|T < 7333415007620259840}]
I20260926 01:02:03.351413 16907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 471d9cef009a492499c7529e853dadbb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "471d9cef009a492499c7529e853dadbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "471d9cef009a492499c7529e853dadbb" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 36547 } } peers { permanent_uuid: "78895aa7bd36435b8a570657fd5012b9" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 43445 } } peers { permanent_uuid: "7faf29afd8094c79b10aa1d9aacf8de2" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 42689 } } }
I20260926 01:02:03.351481 16907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 471d9cef009a492499c7529e853dadbb [sys.catalog]: This master's current role is: LEADER
I20260926 01:02:03.351830 16921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78895aa7bd36435b8a570657fd5012b9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "471d9cef009a492499c7529e853dadbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "471d9cef009a492499c7529e853dadbb" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 36547 } } peers { permanent_uuid: "78895aa7bd36435b8a570657fd5012b9" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 43445 } } peers { permanent_uuid: "7faf29afd8094c79b10aa1d9aacf8de2" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 42689 } } }
I20260926 01:02:03.351913 16921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78895aa7bd36435b8a570657fd5012b9 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:02:03.352500 16938 mvcc.cc:205] Tried to move back new op lower bound from 7333415007620259840 to 7333415007611023360. Current Snapshot: MvccSnapshot[applied={T|T < 7333415007620259840}]
I20260926 01:02:03.352921 16940 mvcc.cc:205] Tried to move back new op lower bound from 7333415007620259840 to 7333415007611023360. Current Snapshot: MvccSnapshot[applied={T|T < 7333415007620259840}]
I20260926 01:02:03.353477 16936 catalog_manager.cc:1347] Generated new cluster ID: 3ef01d19adc94436bdac77c1dd726a98
I20260926 01:02:03.353526 16936 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:02:03.365764 16936 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:02:03.366515 16936 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:02:03.375905 16936 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 471d9cef009a492499c7529e853dadbb: Generated new TSK 0
I20260926 01:02:03.376147 16936 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:02:04.224656 16904 catalog_manager.cc:1259] Loaded cluster ID: 3ef01d19adc94436bdac77c1dd726a98
I20260926 01:02:04.224733 16904 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7faf29afd8094c79b10aa1d9aacf8de2: loading cluster ID for follower catalog manager: success
I20260926 01:02:04.225498 16904 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7faf29afd8094c79b10aa1d9aacf8de2: acquiring CA information for follower catalog manager: success
I20260926 01:02:04.225759 16904 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7faf29afd8094c79b10aa1d9aacf8de2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 01:02:04.293354 16932 catalog_manager.cc:1259] Loaded cluster ID: 3ef01d19adc94436bdac77c1dd726a98
I20260926 01:02:04.293427 16932 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 78895aa7bd36435b8a570657fd5012b9: loading cluster ID for follower catalog manager: success
I20260926 01:02:04.294322 16932 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 78895aa7bd36435b8a570657fd5012b9: acquiring CA information for follower catalog manager: success
I20260926 01:02:04.294656 16932 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 78895aa7bd36435b8a570657fd5012b9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 01:02:04.427855 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 16659
I20260926 01:02:04.433185 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 16735
I20260926 01:02:04.438081 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 16811
2026-09-26T01:02:04Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2825 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384372073861-3372-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:02:04 dist-test-slave-7v10 krb5kdc[16953](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:02:04 dist-test-slave-7v10 krb5kdc[16953](info): set up 2 sockets
Sep 26 01:02:04 dist-test-slave-7v10 krb5kdc[16953](info): commencing operation
krb5kdc: starting...
W20260926 01:02:05.490111 3372 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Sep 26 01:02:05 dist-test-slave-7v10 krb5kdc[16953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:02:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:02:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.75.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab.
Entry for principal kudu/127.3.75.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab.
I20260926 01:02:05.607988 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384372073861-3372-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384372073861-3372-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384372073861-3372-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:40189
--webserver_interface=127.3.75.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:39699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.75.62:40189
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:02:05.692430 16966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:02:05.692698 16966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:02:05.692737 16966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:02:05.693921 16966 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:02:05.694000 16966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:02:05.694031 16966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:02:05.694067 16966 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:02:05.694103 16966 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:02:05.695749 16966 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:39699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384372073861-3372-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384372073861-3372-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.75.62:40189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:40189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384372073861-3372-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:02:05.696046 16966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:02:05.696300 16966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:02:05.701400 16971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:02:05.701385 16972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:02:05.701494 16966 server_base.cc:1034] running on GCE node
W20260926 01:02:05.701387 16974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:02:05.701926 16966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:02:05.702224 16966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:02:05.703359 16966 hybrid_clock.cc:648] HybridClock initialized: now 1790384525703346 us; error 30 us; skew 500 ppm
Sep 26 01:02:05 dist-test-slave-7v10 krb5kdc[16953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384525, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:02:05.705678 16966 init.cc:377] Logged in from keytab as kudu/127.3.75.62@KRBTEST.COM (short username kudu)
I20260926 01:02:05.706133 16966 webserver.cc:460] Webserver started at http://127.3.75.62:43965/ using document root <none> and password file <none>
I20260926 01:02:05.706362 16966 fs_manager.cc:359] Metadata directory not provided
I20260926 01:02:05.706408 16966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:02:05.706493 16966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:02:05.707595 16966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384372073861-3372-0/minicluster-data/master-0/data/instance:
uuid: "3b9fd101841e464abaf3cddf2eab2485"
format_stamp: "Formatted at 2026-09-26 01:02:05 on dist-test-slave-7v10"
server_key: "30f4b68a31168e588134197988b281bc"
server_key_iv: "e6c0daa00d7c3ea6117ad5565228bcb7"
server_key_version: "encryptionkey@0"
I20260926 01:02:05.707959 16966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384372073861-3372-0/minicluster-data/master-0/wal/instance:
uuid: "3b9fd101841e464abaf3cddf2eab2485"
format_stamp: "Formatted at 2026-09-26 01:02:05 on dist-test-slave-7v10"
server_key: "30f4b68a31168e588134197988b281bc"
server_key_iv: "e6c0daa00d7c3ea6117ad5565228bcb7"
server_key_version: "encryptionkey@0"
I20260926 01:02:05.709227 16966 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:02:05.709906 16981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:02:05.710089 16966 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:02:05.710180 16966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384372073861-3372-0/minicluster-data/master-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384372073861-3372-0/minicluster-data/master-0/wal
uuid: "3b9fd101841e464abaf3cddf2eab2485"
format_stamp: "Formatted at 2026-09-26 01:02:05 on dist-test-slave-7v10"
server_key: "30f4b68a31168e588134197988b281bc"
server_key_iv: "e6c0daa00d7c3ea6117ad5565228bcb7"
server_key_version: "encryptionkey@0"
I20260926 01:02:05.710253 16966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384372073861-3372-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384372073861-3372-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384372073861-3372-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:02:05.723109 16966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:02:05.724942 16966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:02:05.725100 16966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:02:05.729254 16966 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.62:40189
I20260926 01:02:05.729317 17034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.62:40189 every 8 connection(s)
I20260926 01:02:05.729676 16966 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
I20260926 01:02:05.730571 17035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:02:05.732084 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 16966
I20260926 01:02:05.732165 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384372073861-3372-0/minicluster-data/master-0/wal/instance
I20260926 01:02:05.732347 3372 external_mini_cluster.cc:1428] Setting key 1ade9ca01b3ca472ab1e3353a298ab96
I20260926 01:02:05.733165 17035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b9fd101841e464abaf3cddf2eab2485: Bootstrap starting.
I20260926 01:02:05.733844 17035 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b9fd101841e464abaf3cddf2eab2485: Neither blocks nor log segments found. Creating new log.
I20260926 01:02:05.734184 17035 log.cc:824] T 00000000000000000000000000000000 P 3b9fd101841e464abaf3cddf2eab2485: Log is configured to *not* fsync() on all Append() calls
I20260926 01:02:05.734949 17035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b9fd101841e464abaf3cddf2eab2485: No bootstrap required, opened a new log
I20260926 01:02:05.737192 17035 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b9fd101841e464abaf3cddf2eab2485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b9fd101841e464abaf3cddf2eab2485" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 40189 } }
I20260926 01:02:05.737396 17035 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b9fd101841e464abaf3cddf2eab2485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:02:05.737466 17035 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b9fd101841e464abaf3cddf2eab2485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b9fd101841e464abaf3cddf2eab2485, State: Initialized, Role: FOLLOWER
I20260926 01:02:05.737607 17035 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b9fd101841e464abaf3cddf2eab2485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b9fd101841e464abaf3cddf2eab2485" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 40189 } }
I20260926 01:02:05.737699 17035 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b9fd101841e464abaf3cddf2eab2485 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:02:05.737731 17035 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b9fd101841e464abaf3cddf2eab2485 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:02:05.737790 17035 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b9fd101841e464abaf3cddf2eab2485 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:02:05.738569 17035 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b9fd101841e464abaf3cddf2eab2485 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b9fd101841e464abaf3cddf2eab2485" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 40189 } }
I20260926 01:02:05.738690 17035 leader_election.cc:302] T 00000000000000000000000000000000 P 3b9fd101841e464abaf3cddf2eab2485 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b9fd101841e464abaf3cddf2eab2485; no voters:
I20260926 01:02:05.738864 17035 leader_election.cc:288] T 00000000000000000000000000000000 P 3b9fd101841e464abaf3cddf2eab2485 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:02:05.738931 17040 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b9fd101841e464abaf3cddf2eab2485 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:02:05.739085 17040 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b9fd101841e464abaf3cddf2eab2485 [term 1 LEADER]: Becoming Leader. State: Replica: 3b9fd101841e464abaf3cddf2eab2485, State: Running, Role: LEADER
I20260926 01:02:05.739162 17035 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b9fd101841e464abaf3cddf2eab2485 [sys.catalog]: configured and running, proceeding with master startup.
Sep 26 01:02:05 dist-test-slave-7v10 krb5kdc[16953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 01:02:05.739176 17040 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b9fd101841e464abaf3cddf2eab2485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b9fd101841e464abaf3cddf2eab2485" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 40189 } }
I20260926 01:02:05.741276 17041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b9fd101841e464abaf3cddf2eab2485 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b9fd101841e464abaf3cddf2eab2485. Latest consensus state: current_term: 1 leader_uuid: "3b9fd101841e464abaf3cddf2eab2485" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b9fd101841e464abaf3cddf2eab2485" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 40189 } } }
I20260926 01:02:05.741297 17042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b9fd101841e464abaf3cddf2eab2485 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b9fd101841e464abaf3cddf2eab2485" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b9fd101841e464abaf3cddf2eab2485" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 40189 } } }
I20260926 01:02:05.741370 17041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b9fd101841e464abaf3cddf2eab2485 [sys.catalog]: This master's current role is: LEADER
I20260926 01:02:05.741384 17042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b9fd101841e464abaf3cddf2eab2485 [sys.catalog]: This master's current role is: LEADER
I20260926 01:02:05.741585 17054 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:02:05.742062 17054 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:02:05.743716 17038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:05.732848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35130 (local address 127.3.75.62:40189)
0926 01:02:05.733014 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:05.733022 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:05.733046 (+ 24us) server_negotiation.cc:407] Connection header received
0926 01:02:05.733498 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:05.733504 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:05.733552 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:05.733665 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:02:05.734515 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:05.735537 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:05.736609 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:05.736987 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:05.737063 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:05.740003 (+ 2940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:02:05.740006 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:02:05.740007 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:02:05.740018 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:02:05.742088 (+ 2070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:05.742515 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:05.742518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:05.742518 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:05.742607 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:05.742850 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:05.742851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:05.742852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:05.743125 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:02:05.743275 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:05.743380 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 01:02:05.743518 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":78,"threads_started":1}
I20260926 01:02:05.743713 17054 catalog_manager.cc:1347] Generated new cluster ID: 77cced8be9fd4a54be508787974dd28a
I20260926 01:02:05.743791 17054 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:02:05.745615 3372 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:02:05.754611 17054 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:02:05.755366 17054 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:02:05.760447 17054 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b9fd101841e464abaf3cddf2eab2485: Generated new TSK 0
I20260926 01:02:05.760618 17054 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:02:05.799808 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 16966
2026-09-26T01:02:05Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1360 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:02:05 dist-test-slave-7v10 krb5kdc[17062](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:02:05 dist-test-slave-7v10 krb5kdc[17062](info): set up 2 sockets
Sep 26 01:02:05 dist-test-slave-7v10 krb5kdc[17062](info): commencing operation
krb5kdc: starting...
W20260926 01:02:06.835016 3372 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Sep 26 01:02:06 dist-test-slave-7v10 krb5kdc[17062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:02:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:02:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.75.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab.
Entry for principal kudu/127.3.75.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab.
I20260926 01:02:06.951958 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:34273
--webserver_interface=127.3.75.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:37523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.75.62:34273
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:02:07.030267 17075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:02:07.030465 17075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:02:07.030488 17075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:02:07.031777 17075 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:02:07.031822 17075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:02:07.031837 17075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:02:07.031850 17075 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:02:07.031862 17075 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:02:07.033301 17075 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:37523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.75.62:34273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:34273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:02:07.033504 17075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:02:07.033776 17075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:02:07.038980 17080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:02:07.039062 17075 server_base.cc:1034] running on GCE node
W20260926 01:02:07.038992 17081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:02:07.038992 17083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:02:07.039451 17075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:02:07.039791 17075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:02:07.040953 17075 hybrid_clock.cc:648] HybridClock initialized: now 1790384527040950 us; error 35 us; skew 500 ppm
Sep 26 01:02:07 dist-test-slave-7v10 krb5kdc[17062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384527, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:02:07.043417 17075 init.cc:377] Logged in from keytab as kudu/127.3.75.62@KRBTEST.COM (short username kudu)
I20260926 01:02:07.043880 17075 webserver.cc:460] Webserver started at http://127.3.75.62:37489/ using document root <none> and password file <none>
I20260926 01:02:07.044106 17075 fs_manager.cc:359] Metadata directory not provided
I20260926 01:02:07.044173 17075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:02:07.044334 17075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:02:07.045382 17075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/master-0/data/instance:
uuid: "aff9338d776a42d18e1ae239f17fa778"
format_stamp: "Formatted at 2026-09-26 01:02:07 on dist-test-slave-7v10"
server_key: "560ae9ec39a670f2ef4d5ee4c143b20e"
server_key_iv: "880158daa51a878aaa41ad79645afad3"
server_key_version: "encryptionkey@0"
I20260926 01:02:07.045737 17075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/master-0/wal/instance:
uuid: "aff9338d776a42d18e1ae239f17fa778"
format_stamp: "Formatted at 2026-09-26 01:02:07 on dist-test-slave-7v10"
server_key: "560ae9ec39a670f2ef4d5ee4c143b20e"
server_key_iv: "880158daa51a878aaa41ad79645afad3"
server_key_version: "encryptionkey@0"
I20260926 01:02:07.047035 17075 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 01:02:07.047780 17090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:02:07.047971 17075 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:02:07.048069 17075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/master-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/master-0/wal
uuid: "aff9338d776a42d18e1ae239f17fa778"
format_stamp: "Formatted at 2026-09-26 01:02:07 on dist-test-slave-7v10"
server_key: "560ae9ec39a670f2ef4d5ee4c143b20e"
server_key_iv: "880158daa51a878aaa41ad79645afad3"
server_key_version: "encryptionkey@0"
I20260926 01:02:07.048156 17075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:02:07.072089 17075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:02:07.074488 17075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:02:07.074653 17075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:02:07.078544 17075 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.62:34273
I20260926 01:02:07.078583 17143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.62:34273 every 8 connection(s)
I20260926 01:02:07.079010 17075 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
I20260926 01:02:07.080042 17144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:02:07.082342 17144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778: Bootstrap starting.
I20260926 01:02:07.082849 17144 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778: Neither blocks nor log segments found. Creating new log.
I20260926 01:02:07.083137 17144 log.cc:824] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778: Log is configured to *not* fsync() on all Append() calls
I20260926 01:02:07.083762 17144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778: No bootstrap required, opened a new log
I20260926 01:02:07.085258 17144 raft_consensus.cc:348] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff9338d776a42d18e1ae239f17fa778" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 34273 } }
I20260926 01:02:07.085383 17144 raft_consensus.cc:374] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:02:07.085404 17144 raft_consensus.cc:729] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aff9338d776a42d18e1ae239f17fa778, State: Initialized, Role: FOLLOWER
I20260926 01:02:07.085537 17144 consensus_queue.cc:260] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff9338d776a42d18e1ae239f17fa778" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 34273 } }
I20260926 01:02:07.085614 17144 raft_consensus.cc:388] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:02:07.085636 17144 raft_consensus.cc:482] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:02:07.085662 17144 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:02:07.086359 17144 raft_consensus.cc:504] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff9338d776a42d18e1ae239f17fa778" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 34273 } }
I20260926 01:02:07.086475 17144 leader_election.cc:302] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aff9338d776a42d18e1ae239f17fa778; no voters:
I20260926 01:02:07.086606 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 17075
I20260926 01:02:07.086639 17144 leader_election.cc:288] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:02:07.086710 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/master-0/wal/instance
I20260926 01:02:07.086691 17147 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:02:07.086843 17147 raft_consensus.cc:686] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778 [term 1 LEADER]: Becoming Leader. State: Replica: aff9338d776a42d18e1ae239f17fa778, State: Running, Role: LEADER
I20260926 01:02:07.086911 17144 sys_catalog.cc:564] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:02:07.086920 3372 external_mini_cluster.cc:1428] Setting key 7c20c3c6138c5ad8c56774ceeb699824
I20260926 01:02:07.087009 17147 consensus_queue.cc:237] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff9338d776a42d18e1ae239f17fa778" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 34273 } }
I20260926 01:02:07.087356 17148 sys_catalog.cc:455] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aff9338d776a42d18e1ae239f17fa778" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff9338d776a42d18e1ae239f17fa778" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 34273 } } }
I20260926 01:02:07.087347 17147 sys_catalog.cc:455] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aff9338d776a42d18e1ae239f17fa778. Latest consensus state: current_term: 1 leader_uuid: "aff9338d776a42d18e1ae239f17fa778" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff9338d776a42d18e1ae239f17fa778" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 34273 } } }
I20260926 01:02:07.087495 17148 sys_catalog.cc:458] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778 [sys.catalog]: This master's current role is: LEADER
I20260926 01:02:07.087507 17147 sys_catalog.cc:458] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778 [sys.catalog]: This master's current role is: LEADER
I20260926 01:02:07.087745 17158 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:02:07.088246 17158 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:02:07.089986 17158 catalog_manager.cc:1347] Generated new cluster ID: aef9d48a0bd14f528e23b4429a7e4d14
I20260926 01:02:07.090035 17158 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 01:02:07 dist-test-slave-7v10 krb5kdc[17062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 01:02:07.099113 17156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:07.087486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33052 (local address 127.3.75.62:34273)
0926 01:02:07.088704 (+ 1218us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:07.088713 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:07.088741 (+ 28us) server_negotiation.cc:407] Connection header received
0926 01:02:07.088862 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:07.088868 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:07.088921 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:07.089049 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:02:07.089694 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:07.091043 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:07.092140 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:07.092734 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:07.092827 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:07.095510 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:02:07.095513 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:02:07.095514 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:02:07.095526 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:02:07.097449 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:07.097854 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:07.097858 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:07.097858 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:07.097985 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:07.098185 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:07.098186 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:07.098187 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:07.098488 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:02:07.098646 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:07.098712 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 01:02:07.098858 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1161,"thread_start_us":71,"threads_started":1}
I20260926 01:02:07.123742 17158 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:02:07.124289 17158 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:02:07.132582 17158 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778: Generated new TSK 0
I20260926 01:02:07.132781 17158 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.75.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab.
Entry for principal kudu/127.3.75.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab.
I20260926 01:02:07.158820 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.1:0
--local_ip_for_outbound_sockets=127.3.75.1
--webserver_interface=127.3.75.1
--webserver_port=0
--tserver_master_addrs=127.3.75.62:34273
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab
--principal=kudu/127.3.75.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:37523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:02:07.239603 17170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:02:07.239809 17170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:02:07.239833 17170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:02:07.240914 17170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:02:07.240947 17170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:02:07.240972 17170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.1
I20260926 01:02:07.242312 17170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:37523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab
--principal=kudu/127.3.75.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.75.1
--webserver_port=0
--tserver_master_addrs=127.3.75.62:34273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.1
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:02:07.242486 17170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:02:07.242655 17170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:02:07.248082 17175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:02:07.248071 17178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:02:07.248076 17176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:02:07.248158 17170 server_base.cc:1034] running on GCE node
I20260926 01:02:07.248476 17170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:02:07.248730 17170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:02:07.249929 17170 hybrid_clock.cc:648] HybridClock initialized: now 1790384527249908 us; error 36 us; skew 500 ppm
Sep 26 01:02:07 dist-test-slave-7v10 krb5kdc[17062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384527, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:02:07.252508 17170 init.cc:377] Logged in from keytab as kudu/127.3.75.1@KRBTEST.COM (short username kudu)
I20260926 01:02:07.252966 17170 webserver.cc:460] Webserver started at http://127.3.75.1:36707/ using document root <none> and password file <none>
I20260926 01:02:07.253202 17170 fs_manager.cc:359] Metadata directory not provided
I20260926 01:02:07.253271 17170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:02:07.253410 17170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:02:07.254464 17170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-0/data/instance:
uuid: "ceb466341f8246e990249e49378d40ba"
format_stamp: "Formatted at 2026-09-26 01:02:07 on dist-test-slave-7v10"
server_key: "ad543bae962dbd335f350cfbe0aa0e1b"
server_key_iv: "4545a50ad287114ad5d185b87cc89683"
server_key_version: "encryptionkey@0"
I20260926 01:02:07.254835 17170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-0/wal/instance:
uuid: "ceb466341f8246e990249e49378d40ba"
format_stamp: "Formatted at 2026-09-26 01:02:07 on dist-test-slave-7v10"
server_key: "ad543bae962dbd335f350cfbe0aa0e1b"
server_key_iv: "4545a50ad287114ad5d185b87cc89683"
server_key_version: "encryptionkey@0"
I20260926 01:02:07.256176 17170 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:02:07.256883 17185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:02:07.257061 17170 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 01:02:07.257158 17170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-0/wal
uuid: "ceb466341f8246e990249e49378d40ba"
format_stamp: "Formatted at 2026-09-26 01:02:07 on dist-test-slave-7v10"
server_key: "ad543bae962dbd335f350cfbe0aa0e1b"
server_key_iv: "4545a50ad287114ad5d185b87cc89683"
server_key_version: "encryptionkey@0"
I20260926 01:02:07.257243 17170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:02:07.278820 17170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:02:07.280813 17170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:02:07.280984 17170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:02:07.281235 17170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:02:07.281618 17170 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:02:07.281673 17170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:02:07.281709 17170 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:02:07.281741 17170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:02:07.287608 17170 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.1:43553
I20260926 01:02:07.287674 17299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.1:43553 every 8 connection(s)
I20260926 01:02:07.288100 17170 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
I20260926 01:02:07.293540 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 17170
I20260926 01:02:07.293658 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-0/wal/instance
Sep 26 01:02:07 dist-test-slave-7v10 krb5kdc[17062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384527, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.1@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 01:02:07.293838 3372 external_mini_cluster.cc:1428] Setting key 877e1184bc079719751f26d1ca802431
I20260926 01:02:07.297190 17156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:07.288833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:58075 (local address 127.3.75.62:34273)
0926 01:02:07.288886 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:07.288888 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:07.289066 (+ 178us) server_negotiation.cc:407] Connection header received
0926 01:02:07.289557 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:07.289558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:07.289572 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:07.289622 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:02:07.290502 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:07.291082 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:07.292007 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:07.292215 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:07.292264 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:07.294348 (+ 2084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:02:07.294350 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:02:07.294350 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:02:07.294357 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:02:07.295825 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:07.296219 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:07.296221 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:07.296221 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:07.296286 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:07.296547 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:07.296548 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:07.296548 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:07.296663 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:02:07.296738 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:07.297013 (+ 275us) server_negotiation.cc:299] Negotiation successful
0926 01:02:07.297102 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 01:02:07.297366 17302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:07.288812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34273 (local address 127.3.75.1:58075)
0926 01:02:07.288959 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:07.288994 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 01:02:07.289470 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:02:07.289676 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:02:07.289679 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:02:07.290231 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:02:07.290425 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:07.290429 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:07.291158 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:07.291159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:07.291937 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:07.291939 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:07.292286 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:07.292287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:07.292397 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:07.292802 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:02:07.292810 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:02:07.294264 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:02:07.295897 (+ 1633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:02:07.295905 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:02:07.295907 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:02:07.296133 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:02:07.296354 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:02:07.296355 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:02:07.296355 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:02:07.296468 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:02:07.296730 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:02:07.296737 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:02:07.296877 (+ 140us) client_negotiation.cc:769] Sending connection context
0926 01:02:07.296978 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 01:02:07.297131 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":84,"threads_started":1}
I20260926 01:02:07.297941 17300 heartbeater.cc:347] Connected to a master server at 127.3.75.62:34273
I20260926 01:02:07.298018 17300 heartbeater.cc:464] Registering TS with master...
I20260926 01:02:07.298236 17300 heartbeater.cc:511] Master 127.3.75.62:34273 requested a full tablet report, sending...
I20260926 01:02:07.298763 17107 ts_manager.cc:194] Registered new tserver with Master: ceb466341f8246e990249e49378d40ba (127.3.75.1:43553)
I20260926 01:02:07.299386 17107 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.75.1@KRBTEST.COM'} at 127.3.75.1:58075
WARNING: no policy specified for kudu/127.3.75.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab.
Entry for principal kudu/127.3.75.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab.
I20260926 01:02:07.318254 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.2:0
--local_ip_for_outbound_sockets=127.3.75.2
--webserver_interface=127.3.75.2
--webserver_port=0
--tserver_master_addrs=127.3.75.62:34273
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab
--principal=kudu/127.3.75.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:37523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:02:07.396817 17305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:02:07.397006 17305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:02:07.397028 17305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:02:07.398072 17305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:02:07.398102 17305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:02:07.398126 17305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.2
I20260926 01:02:07.399524 17305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:37523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab
--principal=kudu/127.3.75.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.75.2
--webserver_port=0
--tserver_master_addrs=127.3.75.62:34273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.2
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:02:07.399706 17305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:02:07.399869 17305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:02:07.404847 17313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:02:07.404876 17310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:02:07.404868 17305 server_base.cc:1034] running on GCE node
W20260926 01:02:07.404843 17311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:02:07.405295 17305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:02:07.405547 17305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:02:07.406685 17305 hybrid_clock.cc:648] HybridClock initialized: now 1790384527406661 us; error 42 us; skew 500 ppm
Sep 26 01:02:07 dist-test-slave-7v10 krb5kdc[17062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384527, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:02:07.409237 17305 init.cc:377] Logged in from keytab as kudu/127.3.75.2@KRBTEST.COM (short username kudu)
I20260926 01:02:07.409704 17305 webserver.cc:460] Webserver started at http://127.3.75.2:36831/ using document root <none> and password file <none>
I20260926 01:02:07.409940 17305 fs_manager.cc:359] Metadata directory not provided
I20260926 01:02:07.410003 17305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:02:07.410140 17305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:02:07.411248 17305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-1/data/instance:
uuid: "3c4cb3d7c2c749558a80ee3ad051c341"
format_stamp: "Formatted at 2026-09-26 01:02:07 on dist-test-slave-7v10"
server_key: "341c439563262cc3d449fe7d9c373127"
server_key_iv: "1855dd36ad1a53a271802b4076d2b444"
server_key_version: "encryptionkey@0"
I20260926 01:02:07.411616 17305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-1/wal/instance:
uuid: "3c4cb3d7c2c749558a80ee3ad051c341"
format_stamp: "Formatted at 2026-09-26 01:02:07 on dist-test-slave-7v10"
server_key: "341c439563262cc3d449fe7d9c373127"
server_key_iv: "1855dd36ad1a53a271802b4076d2b444"
server_key_version: "encryptionkey@0"
I20260926 01:02:07.412925 17305 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:02:07.413688 17320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:02:07.413868 17305 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:02:07.413965 17305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-1/wal
uuid: "3c4cb3d7c2c749558a80ee3ad051c341"
format_stamp: "Formatted at 2026-09-26 01:02:07 on dist-test-slave-7v10"
server_key: "341c439563262cc3d449fe7d9c373127"
server_key_iv: "1855dd36ad1a53a271802b4076d2b444"
server_key_version: "encryptionkey@0"
I20260926 01:02:07.414041 17305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:02:07.424598 17305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:02:07.426975 17305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:02:07.427130 17305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:02:07.427369 17305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:02:07.427678 17305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:02:07.427729 17305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:02:07.427771 17305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:02:07.427793 17305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:02:07.433568 17305 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.2:34311
I20260926 01:02:07.433609 17434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.2:34311 every 8 connection(s)
I20260926 01:02:07.433928 17305 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
Sep 26 01:02:07 dist-test-slave-7v10 krb5kdc[17062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384527, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.2@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 01:02:07.442565 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 17305
I20260926 01:02:07.442679 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-1/wal/instance
I20260926 01:02:07.442734 17156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:07.434633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:34249 (local address 127.3.75.62:34273)
0926 01:02:07.434678 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:07.434680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:07.434914 (+ 234us) server_negotiation.cc:407] Connection header received
0926 01:02:07.435400 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:07.435401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:07.435410 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:07.435507 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:02:07.436408 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:07.436906 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:07.437864 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:07.438046 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:07.438097 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:07.440328 (+ 2231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:02:07.440329 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:02:07.440329 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:02:07.440336 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:02:07.441518 (+ 1182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:07.441872 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:07.441873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:07.441873 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:07.441920 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:07.442120 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:07.442120 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:07.442121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:07.442219 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:02:07.442267 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:07.442523 (+ 256us) server_negotiation.cc:299] Negotiation successful
0926 01:02:07.442612 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 01:02:07.442878 17437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:07.434591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34273 (local address 127.3.75.2:34249)
0926 01:02:07.434745 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:07.434787 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 01:02:07.435316 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:02:07.435560 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:02:07.435564 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:02:07.436125 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:02:07.436327 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:07.436332 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:07.436988 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:07.436989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:07.437789 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:07.437791 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:07.438118 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:07.438118 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:07.438216 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:07.438769 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:02:07.438778 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:02:07.440230 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:02:07.441590 (+ 1360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:02:07.441596 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:02:07.441597 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:02:07.441809 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:02:07.441977 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:02:07.441977 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:02:07.441977 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:02:07.442069 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:02:07.442278 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:02:07.442282 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:02:07.442394 (+ 112us) client_negotiation.cc:769] Sending connection context
0926 01:02:07.442474 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 01:02:07.442593 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":85,"threads_started":1}
I20260926 01:02:07.442916 3372 external_mini_cluster.cc:1428] Setting key 1e3669bf490c06e9fe63d457b61d1b0d
I20260926 01:02:07.443506 17435 heartbeater.cc:347] Connected to a master server at 127.3.75.62:34273
I20260926 01:02:07.443629 17435 heartbeater.cc:464] Registering TS with master...
I20260926 01:02:07.443847 17435 heartbeater.cc:511] Master 127.3.75.62:34273 requested a full tablet report, sending...
I20260926 01:02:07.444260 17107 ts_manager.cc:194] Registered new tserver with Master: 3c4cb3d7c2c749558a80ee3ad051c341 (127.3.75.2:34311)
I20260926 01:02:07.444641 17107 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.75.2@KRBTEST.COM'} at 127.3.75.2:34249
WARNING: no policy specified for kudu/127.3.75.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab.
Entry for principal kudu/127.3.75.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab.
I20260926 01:02:07.467489 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.3:0
--local_ip_for_outbound_sockets=127.3.75.3
--webserver_interface=127.3.75.3
--webserver_port=0
--tserver_master_addrs=127.3.75.62:34273
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab
--principal=kudu/127.3.75.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:37523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:02:07.547891 17440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:02:07.548090 17440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:02:07.548117 17440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:02:07.549258 17440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:02:07.549301 17440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:02:07.549328 17440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.3
I20260926 01:02:07.550863 17440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:37523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab
--principal=kudu/127.3.75.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.75.3
--webserver_port=0
--tserver_master_addrs=127.3.75.62:34273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.3
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:02:07.551147 17440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:02:07.551384 17440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:02:07.556223 17445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:02:07.556260 17448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:02:07.556470 17440 server_base.cc:1034] running on GCE node
W20260926 01:02:07.556274 17446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:02:07.556711 17440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:02:07.556985 17440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:02:07.558149 17440 hybrid_clock.cc:648] HybridClock initialized: now 1790384527558133 us; error 29 us; skew 500 ppm
Sep 26 01:02:07 dist-test-slave-7v10 krb5kdc[17062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384527, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:02:07.560695 17440 init.cc:377] Logged in from keytab as kudu/127.3.75.3@KRBTEST.COM (short username kudu)
I20260926 01:02:07.561069 17440 webserver.cc:460] Webserver started at http://127.3.75.3:32939/ using document root <none> and password file <none>
I20260926 01:02:07.561309 17440 fs_manager.cc:359] Metadata directory not provided
I20260926 01:02:07.561354 17440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:02:07.561496 17440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:02:07.562450 17440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-2/data/instance:
uuid: "075f2a85cca14f3c892ecaeba6244ea4"
format_stamp: "Formatted at 2026-09-26 01:02:07 on dist-test-slave-7v10"
server_key: "82af5ba29d9171d4f288cc330359ee73"
server_key_iv: "3dd837ed4adcb4be2e765588e539c9f3"
server_key_version: "encryptionkey@0"
I20260926 01:02:07.562741 17440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-2/wal/instance:
uuid: "075f2a85cca14f3c892ecaeba6244ea4"
format_stamp: "Formatted at 2026-09-26 01:02:07 on dist-test-slave-7v10"
server_key: "82af5ba29d9171d4f288cc330359ee73"
server_key_iv: "3dd837ed4adcb4be2e765588e539c9f3"
server_key_version: "encryptionkey@0"
I20260926 01:02:07.564023 17440 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:02:07.564651 17455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:02:07.564797 17440 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 01:02:07.564882 17440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-2/wal
uuid: "075f2a85cca14f3c892ecaeba6244ea4"
format_stamp: "Formatted at 2026-09-26 01:02:07 on dist-test-slave-7v10"
server_key: "82af5ba29d9171d4f288cc330359ee73"
server_key_iv: "3dd837ed4adcb4be2e765588e539c9f3"
server_key_version: "encryptionkey@0"
I20260926 01:02:07.564934 17440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:02:07.589928 17440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:02:07.592229 17440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:02:07.592350 17440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:02:07.592519 17440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:02:07.592833 17440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:02:07.592870 17440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:02:07.592913 17440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:02:07.592941 17440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:02:07.598563 17440 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.3:39451
I20260926 01:02:07.598657 17572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.3:39451 every 8 connection(s)
I20260926 01:02:07.598981 17440 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
I20260926 01:02:07.602005 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 17440
I20260926 01:02:07.602099 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-2/wal/instance
I20260926 01:02:07.602299 3372 external_mini_cluster.cc:1428] Setting key a8857188b7bb5bfed8a2e6192973c459
Sep 26 01:02:07 dist-test-slave-7v10 krb5kdc[17062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384527, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.3@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 01:02:07.608225 17156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:07.599705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.3:44677 (local address 127.3.75.62:34273)
0926 01:02:07.599746 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:07.599749 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:07.599933 (+ 184us) server_negotiation.cc:407] Connection header received
0926 01:02:07.600447 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:07.600448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:07.600460 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:07.600526 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:02:07.601469 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:07.602109 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:07.603544 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:07.603732 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:07.603793 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:07.605727 (+ 1934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:02:07.605729 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:02:07.605729 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:02:07.605736 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:02:07.607138 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:07.607485 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:07.607486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:07.607486 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:07.607534 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:07.607734 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:07.607735 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:07.607735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:07.607832 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:02:07.607880 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:07.608103 (+ 223us) server_negotiation.cc:299] Negotiation successful
0926 01:02:07.608162 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 01:02:07.608385 17575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:07.599681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34273 (local address 127.3.75.3:44677)
0926 01:02:07.599810 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:07.599852 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 01:02:07.600346 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:02:07.600597 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:02:07.600601 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:02:07.601180 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:02:07.601373 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:07.601378 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:07.602234 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:07.602235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:07.603475 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:07.603477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:07.603814 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:07.603815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:07.603913 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:07.604322 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:02:07.604330 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:02:07.605648 (+ 1318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:02:07.607202 (+ 1554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:02:07.607207 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:02:07.607209 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:02:07.607420 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:02:07.607588 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:02:07.607588 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:02:07.607589 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:02:07.607682 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:02:07.607883 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:02:07.607887 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:02:07.607997 (+ 110us) client_negotiation.cc:769] Sending connection context
0926 01:02:07.608076 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 01:02:07.608190 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":60,"threads_started":1}
I20260926 01:02:07.608870 17573 heartbeater.cc:347] Connected to a master server at 127.3.75.62:34273
I20260926 01:02:07.608978 17573 heartbeater.cc:464] Registering TS with master...
I20260926 01:02:07.609155 17573 heartbeater.cc:511] Master 127.3.75.62:34273 requested a full tablet report, sending...
I20260926 01:02:07.609529 17107 ts_manager.cc:194] Registered new tserver with Master: 075f2a85cca14f3c892ecaeba6244ea4 (127.3.75.3:39451)
I20260926 01:02:07.609895 17107 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.75.3@KRBTEST.COM'} at 127.3.75.3:44677
I20260926 01:02:07.614051 3372 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 01:02:07.620854 17156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:07.615330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33062 (local address 127.3.75.62:34273)
0926 01:02:07.615392 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:07.615394 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:07.615473 (+ 79us) server_negotiation.cc:407] Connection header received
0926 01:02:07.615506 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:07.615506 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:07.615514 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:07.615577 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:02:07.616087 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:07.616543 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:07.617291 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:07.617458 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:07.617506 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:07.618641 (+ 1135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:02:07.618642 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:02:07.618642 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:02:07.618647 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:02:07.619952 (+ 1305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:07.620240 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:07.620241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:07.620242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:07.620295 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:07.620449 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:07.620450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:07.620450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:07.620571 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:02:07.620639 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:07.620722 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 01:02:07.620774 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 01:02:07.622330 17107 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33062:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:02:07.624013 17107 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:02:07.629854 17584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:07.626196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58546 (local address 127.3.75.1:43553)
0926 01:02:07.626493 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:07.626498 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:07.626509 (+ 11us) server_negotiation.cc:407] Connection header received
0926 01:02:07.626597 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:07.626599 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:07.626616 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:07.626683 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:02:07.626894 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:07.627706 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:07.628754 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:07.629405 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:07.629503 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:07.629580 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:07.629733 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 01:02:07.629776 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":87,"threads_started":1}
I20260926 01:02:07.629845 17582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:07.626071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.1:43553 (local address 127.0.0.1:58546)
0926 01:02:07.626380 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:07.626423 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 01:02:07.626490 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:02:07.626761 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:02:07.626762 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:02:07.626763 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:02:07.626808 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:07.626809 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:07.627806 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:07.627807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:07.628647 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:07.628649 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:07.629504 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:07.629505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:07.629612 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:07.629625 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 01:02:07.629705 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 01:02:07.629761 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":57,"threads_started":1}
I20260926 01:02:07.630270 17581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:07.626270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.3:39451 (local address 127.0.0.1:50934)
0926 01:02:07.626564 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:07.626572 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 01:02:07.626612 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:02:07.626938 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:02:07.626939 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:02:07.626942 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:02:07.626992 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:07.626994 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:07.627899 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:07.627900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:07.629052 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:07.629053 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:07.630023 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:07.630024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:07.630082 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:07.630086 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:02:07.630143 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 01:02:07.630185 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":55,"threads_started":1}
I20260926 01:02:07.630304 17586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:07.626469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50934 (local address 127.3.75.3:39451)
0926 01:02:07.626657 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:07.626662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:07.626672 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:02:07.626704 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:07.626706 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:07.626719 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:07.626785 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:02:07.627060 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:07.627791 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:07.629133 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:07.629878 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:07.629944 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:07.629991 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:07.630173 (+ 182us) server_negotiation.cc:299] Negotiation successful
0926 01:02:07.630218 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":87,"threads_started":1}
I20260926 01:02:07.630961 17233 tablet_service.cc:1467] Processing CreateTablet for tablet 568ac290961248cab8904dcdea6c6167 (DEFAULT_TABLE table=test-table [id=2409715a68f749fcbbfe088b0fd29a88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:02:07.631112 17503 tablet_service.cc:1467] Processing CreateTablet for tablet 568ac290961248cab8904dcdea6c6167 (DEFAULT_TABLE table=test-table [id=2409715a68f749fcbbfe088b0fd29a88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:02:07.631300 17585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:07.626055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.2:34311 (local address 127.0.0.1:35868)
0926 01:02:07.626373 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:07.626428 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 01:02:07.626491 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:02:07.626731 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:02:07.626734 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:02:07.626737 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:02:07.626803 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:07.626805 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:07.629022 (+ 2217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:07.629023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:07.630102 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:07.630103 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:07.631057 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:07.631058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:07.631119 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:07.631123 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:02:07.631174 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 01:02:07.631219 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":2,"thread_start_us":85,"threads_started":1}
I20260926 01:02:07.631361 17233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 568ac290961248cab8904dcdea6c6167. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:02:07.631376 17503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 568ac290961248cab8904dcdea6c6167. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:02:07.631558 17583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:07.626158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35868 (local address 127.3.75.2:34311)
0926 01:02:07.626311 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:07.626316 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:07.626577 (+ 261us) server_negotiation.cc:407] Connection header received
0926 01:02:07.626626 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:07.626628 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:07.626644 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:07.626719 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:02:07.627823 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:07.628913 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:07.630186 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:07.630951 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:07.631054 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:07.631107 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:07.631456 (+ 349us) server_negotiation.cc:299] Negotiation successful
0926 01:02:07.631500 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":80,"threads_started":1}
I20260926 01:02:07.632107 17368 tablet_service.cc:1467] Processing CreateTablet for tablet 568ac290961248cab8904dcdea6c6167 (DEFAULT_TABLE table=test-table [id=2409715a68f749fcbbfe088b0fd29a88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:02:07.632336 17368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 568ac290961248cab8904dcdea6c6167. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:02:07.634605 17588 tablet_bootstrap.cc:492] T 568ac290961248cab8904dcdea6c6167 P 075f2a85cca14f3c892ecaeba6244ea4: Bootstrap starting.
I20260926 01:02:07.634951 17587 tablet_bootstrap.cc:492] T 568ac290961248cab8904dcdea6c6167 P ceb466341f8246e990249e49378d40ba: Bootstrap starting.
I20260926 01:02:07.635123 17589 tablet_bootstrap.cc:492] T 568ac290961248cab8904dcdea6c6167 P 3c4cb3d7c2c749558a80ee3ad051c341: Bootstrap starting.
I20260926 01:02:07.635354 17588 tablet_bootstrap.cc:654] T 568ac290961248cab8904dcdea6c6167 P 075f2a85cca14f3c892ecaeba6244ea4: Neither blocks nor log segments found. Creating new log.
I20260926 01:02:07.635703 17587 tablet_bootstrap.cc:654] T 568ac290961248cab8904dcdea6c6167 P ceb466341f8246e990249e49378d40ba: Neither blocks nor log segments found. Creating new log.
I20260926 01:02:07.635833 17589 tablet_bootstrap.cc:654] T 568ac290961248cab8904dcdea6c6167 P 3c4cb3d7c2c749558a80ee3ad051c341: Neither blocks nor log segments found. Creating new log.
I20260926 01:02:07.635928 17588 log.cc:824] T 568ac290961248cab8904dcdea6c6167 P 075f2a85cca14f3c892ecaeba6244ea4: Log is configured to *not* fsync() on all Append() calls
I20260926 01:02:07.636363 17587 log.cc:824] T 568ac290961248cab8904dcdea6c6167 P ceb466341f8246e990249e49378d40ba: Log is configured to *not* fsync() on all Append() calls
I20260926 01:02:07.636430 17589 log.cc:824] T 568ac290961248cab8904dcdea6c6167 P 3c4cb3d7c2c749558a80ee3ad051c341: Log is configured to *not* fsync() on all Append() calls
I20260926 01:02:07.636682 17588 tablet_bootstrap.cc:492] T 568ac290961248cab8904dcdea6c6167 P 075f2a85cca14f3c892ecaeba6244ea4: No bootstrap required, opened a new log
I20260926 01:02:07.636771 17588 ts_tablet_manager.cc:1397] T 568ac290961248cab8904dcdea6c6167 P 075f2a85cca14f3c892ecaeba6244ea4: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 01:02:07.637094 17587 tablet_bootstrap.cc:492] T 568ac290961248cab8904dcdea6c6167 P ceb466341f8246e990249e49378d40ba: No bootstrap required, opened a new log
I20260926 01:02:07.637177 17587 ts_tablet_manager.cc:1397] T 568ac290961248cab8904dcdea6c6167 P ceb466341f8246e990249e49378d40ba: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 01:02:07.638373 17588 raft_consensus.cc:348] T 568ac290961248cab8904dcdea6c6167 P 075f2a85cca14f3c892ecaeba6244ea4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4cb3d7c2c749558a80ee3ad051c341" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 34311 } } peers { permanent_uuid: "ceb466341f8246e990249e49378d40ba" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 43553 } } peers { permanent_uuid: "075f2a85cca14f3c892ecaeba6244ea4" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 39451 } }
I20260926 01:02:07.638532 17588 raft_consensus.cc:374] T 568ac290961248cab8904dcdea6c6167 P 075f2a85cca14f3c892ecaeba6244ea4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:02:07.638577 17588 raft_consensus.cc:729] T 568ac290961248cab8904dcdea6c6167 P 075f2a85cca14f3c892ecaeba6244ea4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075f2a85cca14f3c892ecaeba6244ea4, State: Initialized, Role: FOLLOWER
I20260926 01:02:07.638540 17587 raft_consensus.cc:348] T 568ac290961248cab8904dcdea6c6167 P ceb466341f8246e990249e49378d40ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4cb3d7c2c749558a80ee3ad051c341" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 34311 } } peers { permanent_uuid: "ceb466341f8246e990249e49378d40ba" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 43553 } } peers { permanent_uuid: "075f2a85cca14f3c892ecaeba6244ea4" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 39451 } }
I20260926 01:02:07.638689 17588 consensus_queue.cc:260] T 568ac290961248cab8904dcdea6c6167 P 075f2a85cca14f3c892ecaeba6244ea4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4cb3d7c2c749558a80ee3ad051c341" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 34311 } } peers { permanent_uuid: "ceb466341f8246e990249e49378d40ba" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 43553 } } peers { permanent_uuid: "075f2a85cca14f3c892ecaeba6244ea4" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 39451 } }
I20260926 01:02:07.638731 17587 raft_consensus.cc:374] T 568ac290961248cab8904dcdea6c6167 P ceb466341f8246e990249e49378d40ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:02:07.638788 17587 raft_consensus.cc:729] T 568ac290961248cab8904dcdea6c6167 P ceb466341f8246e990249e49378d40ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceb466341f8246e990249e49378d40ba, State: Initialized, Role: FOLLOWER
I20260926 01:02:07.638933 17587 consensus_queue.cc:260] T 568ac290961248cab8904dcdea6c6167 P ceb466341f8246e990249e49378d40ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4cb3d7c2c749558a80ee3ad051c341" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 34311 } } peers { permanent_uuid: "ceb466341f8246e990249e49378d40ba" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 43553 } } peers { permanent_uuid: "075f2a85cca14f3c892ecaeba6244ea4" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 39451 } }
I20260926 01:02:07.639024 17588 ts_tablet_manager.cc:1428] T 568ac290961248cab8904dcdea6c6167 P 075f2a85cca14f3c892ecaeba6244ea4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 01:02:07.639133 17573 heartbeater.cc:503] Master 127.3.75.62:34273 was elected leader, sending a full tablet report...
I20260926 01:02:07.639204 17587 ts_tablet_manager.cc:1428] T 568ac290961248cab8904dcdea6c6167 P ceb466341f8246e990249e49378d40ba: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 01:02:07.639257 17300 heartbeater.cc:503] Master 127.3.75.62:34273 was elected leader, sending a full tablet report...
I20260926 01:02:07.640218 17589 tablet_bootstrap.cc:492] T 568ac290961248cab8904dcdea6c6167 P 3c4cb3d7c2c749558a80ee3ad051c341: No bootstrap required, opened a new log
I20260926 01:02:07.640285 17589 ts_tablet_manager.cc:1397] T 568ac290961248cab8904dcdea6c6167 P 3c4cb3d7c2c749558a80ee3ad051c341: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260926 01:02:07.641701 17589 raft_consensus.cc:348] T 568ac290961248cab8904dcdea6c6167 P 3c4cb3d7c2c749558a80ee3ad051c341 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4cb3d7c2c749558a80ee3ad051c341" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 34311 } } peers { permanent_uuid: "ceb466341f8246e990249e49378d40ba" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 43553 } } peers { permanent_uuid: "075f2a85cca14f3c892ecaeba6244ea4" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 39451 } }
I20260926 01:02:07.641947 17589 raft_consensus.cc:374] T 568ac290961248cab8904dcdea6c6167 P 3c4cb3d7c2c749558a80ee3ad051c341 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:02:07.642004 17589 raft_consensus.cc:729] T 568ac290961248cab8904dcdea6c6167 P 3c4cb3d7c2c749558a80ee3ad051c341 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4cb3d7c2c749558a80ee3ad051c341, State: Initialized, Role: FOLLOWER
I20260926 01:02:07.642141 17589 consensus_queue.cc:260] T 568ac290961248cab8904dcdea6c6167 P 3c4cb3d7c2c749558a80ee3ad051c341 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4cb3d7c2c749558a80ee3ad051c341" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 34311 } } peers { permanent_uuid: "ceb466341f8246e990249e49378d40ba" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 43553 } } peers { permanent_uuid: "075f2a85cca14f3c892ecaeba6244ea4" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 39451 } }
I20260926 01:02:07.642419 17589 ts_tablet_manager.cc:1428] T 568ac290961248cab8904dcdea6c6167 P 3c4cb3d7c2c749558a80ee3ad051c341: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 01:02:07.642426 17435 heartbeater.cc:503] Master 127.3.75.62:34273 was elected leader, sending a full tablet report...
I20260926 01:02:07.651713 17594 raft_consensus.cc:482] T 568ac290961248cab8904dcdea6c6167 P ceb466341f8246e990249e49378d40ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:02:07.651839 17594 raft_consensus.cc:504] T 568ac290961248cab8904dcdea6c6167 P ceb466341f8246e990249e49378d40ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4cb3d7c2c749558a80ee3ad051c341" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 34311 } } peers { permanent_uuid: "ceb466341f8246e990249e49378d40ba" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 43553 } } peers { permanent_uuid: "075f2a85cca14f3c892ecaeba6244ea4" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 39451 } }
I20260926 01:02:07.652225 17594 leader_election.cc:288] T 568ac290961248cab8904dcdea6c6167 P ceb466341f8246e990249e49378d40ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c4cb3d7c2c749558a80ee3ad051c341 (127.3.75.2:34311), 075f2a85cca14f3c892ecaeba6244ea4 (127.3.75.3:39451)
I20260926 01:02:07.655777 17586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:07.652607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:60719 (local address 127.3.75.3:39451)
0926 01:02:07.652664 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:07.652666 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:07.652673 (+ 7us) server_negotiation.cc:407] Connection header received
0926 01:02:07.652695 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:07.652696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:07.652710 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:07.652793 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:02:07.652950 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:07.653578 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:07.654823 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:07.655421 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:07.655489 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:07.655499 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:07.655668 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 01:02:07.655706 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 01:02:07.655787 17583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:07.652435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:46369 (local address 127.3.75.2:34311)
0926 01:02:07.652498 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:07.652500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:07.652549 (+ 49us) server_negotiation.cc:407] Connection header received
0926 01:02:07.652654 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:07.652655 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:07.652667 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:07.652724 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:02:07.652875 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:07.653415 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:07.654823 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:07.655412 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:07.655489 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:07.655499 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:07.655668 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 01:02:07.655706 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 01:02:07.655810 17302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:07.652371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.2:34311 (local address 127.3.75.1:46369)
0926 01:02:07.652486 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:07.652496 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:02:07.652564 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:02:07.652728 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:02:07.652728 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:02:07.652730 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:02:07.652800 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:07.652802 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:07.653524 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:07.653527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:07.654716 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:07.654719 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:07.655496 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:07.655497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:07.655591 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:07.655593 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 01:02:07.655651 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 01:02:07.655696 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"mutex_wait_us":22}
I20260926 01:02:07.655812 17596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:07.652387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.3:39451 (local address 127.3.75.1:60719)
0926 01:02:07.652581 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:07.652590 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:02:07.652632 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:02:07.652802 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:02:07.652804 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:02:07.652805 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:02:07.652877 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:07.652879 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:07.653653 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:07.653653 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:07.654742 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:07.654744 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:07.655496 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:07.655497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:07.655588 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:07.655592 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:02:07.655651 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 01:02:07.655696 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":4,"thread_start_us":130,"threads_started":1}
I20260926 01:02:07.656141 17388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568ac290961248cab8904dcdea6c6167" candidate_uuid: "ceb466341f8246e990249e49378d40ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4cb3d7c2c749558a80ee3ad051c341" is_pre_election: true
I20260926 01:02:07.656281 17388 raft_consensus.cc:2454] T 568ac290961248cab8904dcdea6c6167 P 3c4cb3d7c2c749558a80ee3ad051c341 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ceb466341f8246e990249e49378d40ba in term 0.
I20260926 01:02:07.656329 17524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568ac290961248cab8904dcdea6c6167" candidate_uuid: "ceb466341f8246e990249e49378d40ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075f2a85cca14f3c892ecaeba6244ea4" is_pre_election: true
I20260926 01:02:07.656481 17524 raft_consensus.cc:2454] T 568ac290961248cab8904dcdea6c6167 P 075f2a85cca14f3c892ecaeba6244ea4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ceb466341f8246e990249e49378d40ba in term 0.
I20260926 01:02:07.656505 17187 leader_election.cc:302] T 568ac290961248cab8904dcdea6c6167 P ceb466341f8246e990249e49378d40ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4cb3d7c2c749558a80ee3ad051c341, ceb466341f8246e990249e49378d40ba; no voters:
I20260926 01:02:07.656642 17594 raft_consensus.cc:2781] T 568ac290961248cab8904dcdea6c6167 P ceb466341f8246e990249e49378d40ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:02:07.656708 17594 raft_consensus.cc:482] T 568ac290961248cab8904dcdea6c6167 P ceb466341f8246e990249e49378d40ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:02:07.656726 17594 raft_consensus.cc:3037] T 568ac290961248cab8904dcdea6c6167 P ceb466341f8246e990249e49378d40ba [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:02:07.657420 17594 raft_consensus.cc:504] T 568ac290961248cab8904dcdea6c6167 P ceb466341f8246e990249e49378d40ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4cb3d7c2c749558a80ee3ad051c341" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 34311 } } peers { permanent_uuid: "ceb466341f8246e990249e49378d40ba" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 43553 } } peers { permanent_uuid: "075f2a85cca14f3c892ecaeba6244ea4" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 39451 } }
I20260926 01:02:07.657557 17594 leader_election.cc:288] T 568ac290961248cab8904dcdea6c6167 P ceb466341f8246e990249e49378d40ba [CANDIDATE]: Term 1 election: Requested vote from peers 3c4cb3d7c2c749558a80ee3ad051c341 (127.3.75.2:34311), 075f2a85cca14f3c892ecaeba6244ea4 (127.3.75.3:39451)
I20260926 01:02:07.657696 17524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568ac290961248cab8904dcdea6c6167" candidate_uuid: "ceb466341f8246e990249e49378d40ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075f2a85cca14f3c892ecaeba6244ea4"
I20260926 01:02:07.657692 17388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568ac290961248cab8904dcdea6c6167" candidate_uuid: "ceb466341f8246e990249e49378d40ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4cb3d7c2c749558a80ee3ad051c341"
I20260926 01:02:07.657756 17524 raft_consensus.cc:3037] T 568ac290961248cab8904dcdea6c6167 P 075f2a85cca14f3c892ecaeba6244ea4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:02:07.657793 17388 raft_consensus.cc:3037] T 568ac290961248cab8904dcdea6c6167 P 3c4cb3d7c2c749558a80ee3ad051c341 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:02:07.658442 17524 raft_consensus.cc:2454] T 568ac290961248cab8904dcdea6c6167 P 075f2a85cca14f3c892ecaeba6244ea4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ceb466341f8246e990249e49378d40ba in term 1.
I20260926 01:02:07.658555 17388 raft_consensus.cc:2454] T 568ac290961248cab8904dcdea6c6167 P 3c4cb3d7c2c749558a80ee3ad051c341 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ceb466341f8246e990249e49378d40ba in term 1.
I20260926 01:02:07.658613 17186 leader_election.cc:302] T 568ac290961248cab8904dcdea6c6167 P ceb466341f8246e990249e49378d40ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075f2a85cca14f3c892ecaeba6244ea4, ceb466341f8246e990249e49378d40ba; no voters:
I20260926 01:02:07.658702 17594 raft_consensus.cc:2781] T 568ac290961248cab8904dcdea6c6167 P ceb466341f8246e990249e49378d40ba [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:02:07.658911 17594 raft_consensus.cc:686] T 568ac290961248cab8904dcdea6c6167 P ceb466341f8246e990249e49378d40ba [term 1 LEADER]: Becoming Leader. State: Replica: ceb466341f8246e990249e49378d40ba, State: Running, Role: LEADER
I20260926 01:02:07.659001 17594 consensus_queue.cc:237] T 568ac290961248cab8904dcdea6c6167 P ceb466341f8246e990249e49378d40ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4cb3d7c2c749558a80ee3ad051c341" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 34311 } } peers { permanent_uuid: "ceb466341f8246e990249e49378d40ba" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 43553 } } peers { permanent_uuid: "075f2a85cca14f3c892ecaeba6244ea4" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 39451 } }
I20260926 01:02:07.659984 17106 catalog_manager.cc:5495] T 568ac290961248cab8904dcdea6c6167 P ceb466341f8246e990249e49378d40ba reported cstate change: term changed from 0 to 1, leader changed from <none> to ceb466341f8246e990249e49378d40ba (127.3.75.1). New cstate: current_term: 1 leader_uuid: "ceb466341f8246e990249e49378d40ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4cb3d7c2c749558a80ee3ad051c341" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 34311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ceb466341f8246e990249e49378d40ba" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 43553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "075f2a85cca14f3c892ecaeba6244ea4" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 39451 } health_report { overall_health: UNKNOWN } } }
I20260926 01:02:07.664638 17584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:07.662646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58560 (local address 127.3.75.1:43553)
0926 01:02:07.662680 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:07.662682 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:07.662707 (+ 25us) server_negotiation.cc:407] Connection header received
0926 01:02:07.662766 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:07.662767 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:07.662782 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:07.662843 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:02:07.662983 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:07.663398 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:07.664069 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:07.664239 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:07.664286 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:07.664402 (+ 116us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:02:07.664483 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:02:07.664536 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:07.664566 (+ 30us) server_negotiation.cc:299] Negotiation successful
0926 01:02:07.664579 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 01:02:07.666136 17388 raft_consensus.cc:1261] T 568ac290961248cab8904dcdea6c6167 P 3c4cb3d7c2c749558a80ee3ad051c341 [term 1 FOLLOWER]: Refusing update from remote peer ceb466341f8246e990249e49378d40ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:02:07.666136 17524 raft_consensus.cc:1261] T 568ac290961248cab8904dcdea6c6167 P 075f2a85cca14f3c892ecaeba6244ea4 [term 1 FOLLOWER]: Refusing update from remote peer ceb466341f8246e990249e49378d40ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:02:07.666524 17594 consensus_queue.cc:1035] T 568ac290961248cab8904dcdea6c6167 P ceb466341f8246e990249e49378d40ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "075f2a85cca14f3c892ecaeba6244ea4" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 39451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:02:07.666641 17597 consensus_queue.cc:1035] T 568ac290961248cab8904dcdea6c6167 P ceb466341f8246e990249e49378d40ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4cb3d7c2c749558a80ee3ad051c341" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 34311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:02:07.668694 17600 mvcc.cc:205] Tried to move back new op lower bound from 7333415025318227968 to 7333415025291780096. Current Snapshot: MvccSnapshot[applied={T|T < 7333415025318227968}]
I20260926 01:02:07.669235 17602 mvcc.cc:205] Tried to move back new op lower bound from 7333415025318227968 to 7333415025291780096. Current Snapshot: MvccSnapshot[applied={T|T < 7333415025318227968}]
I20260926 01:02:07.669461 17601 mvcc.cc:205] Tried to move back new op lower bound from 7333415025318227968 to 7333415025291780096. Current Snapshot: MvccSnapshot[applied={T|T < 7333415025318227968 or (T in {7333415025318227968})}]
I20260926 01:02:07.671231 17107 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:02:07.671332 17107 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:02:07.672029 17107 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778: Sending DeleteTablet for 3 replicas of tablet 568ac290961248cab8904dcdea6c6167
I20260926 01:02:07.672379 17233 tablet_service.cc:1514] Processing DeleteTablet for tablet 568ac290961248cab8904dcdea6c6167 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:02:07 UTC) from {username='kudu', principal='kudu/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:58546
I20260926 01:02:07.672408 17503 tablet_service.cc:1514] Processing DeleteTablet for tablet 568ac290961248cab8904dcdea6c6167 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:02:07 UTC) from {username='kudu', principal='kudu/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:50934
I20260926 01:02:07.672408 17368 tablet_service.cc:1514] Processing DeleteTablet for tablet 568ac290961248cab8904dcdea6c6167 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:02:07 UTC) from {username='kudu', principal='kudu/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:35868
I20260926 01:02:07.672623 17612 tablet_replica.cc:330] T 568ac290961248cab8904dcdea6c6167 P 3c4cb3d7c2c749558a80ee3ad051c341: stopping tablet replica
I20260926 01:02:07.672657 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 17170
I20260926 01:02:07.672727 17612 raft_consensus.cc:2229] T 568ac290961248cab8904dcdea6c6167 P 3c4cb3d7c2c749558a80ee3ad051c341 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:02:07.672828 17612 raft_consensus.cc:2258] T 568ac290961248cab8904dcdea6c6167 P 3c4cb3d7c2c749558a80ee3ad051c341 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:02:07.672974 17611 tablet_replica.cc:330] T 568ac290961248cab8904dcdea6c6167 P 075f2a85cca14f3c892ecaeba6244ea4: stopping tablet replica
I20260926 01:02:07.673053 17611 raft_consensus.cc:2229] T 568ac290961248cab8904dcdea6c6167 P 075f2a85cca14f3c892ecaeba6244ea4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:02:07.673347 17611 raft_consensus.cc:2258] T 568ac290961248cab8904dcdea6c6167 P 075f2a85cca14f3c892ecaeba6244ea4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:02:07.673741 17611 ts_tablet_manager.cc:1905] T 568ac290961248cab8904dcdea6c6167 P 075f2a85cca14f3c892ecaeba6244ea4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:02:07.674247 17612 ts_tablet_manager.cc:1905] T 568ac290961248cab8904dcdea6c6167 P 3c4cb3d7c2c749558a80ee3ad051c341: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:02:07.675614 17611 ts_tablet_manager.cc:1918] T 568ac290961248cab8904dcdea6c6167 P 075f2a85cca14f3c892ecaeba6244ea4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:02:07.675715 17611 log.cc:1196] T 568ac290961248cab8904dcdea6c6167 P 075f2a85cca14f3c892ecaeba6244ea4: Deleting WAL directory at /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-2/wal/wals/568ac290961248cab8904dcdea6c6167
I20260926 01:02:07.676030 17611 ts_tablet_manager.cc:1939] T 568ac290961248cab8904dcdea6c6167 P 075f2a85cca14f3c892ecaeba6244ea4: Deleting consensus metadata
I20260926 01:02:07.676186 17612 ts_tablet_manager.cc:1918] T 568ac290961248cab8904dcdea6c6167 P 3c4cb3d7c2c749558a80ee3ad051c341: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:02:07.676265 17612 log.cc:1196] T 568ac290961248cab8904dcdea6c6167 P 3c4cb3d7c2c749558a80ee3ad051c341: Deleting WAL directory at /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-1/wal/wals/568ac290961248cab8904dcdea6c6167
I20260926 01:02:07.676460 17091 catalog_manager.cc:4841] TS 075f2a85cca14f3c892ecaeba6244ea4 (127.3.75.3:39451): tablet 568ac290961248cab8904dcdea6c6167 (table test-table [id=2409715a68f749fcbbfe088b0fd29a88]) successfully deleted
I20260926 01:02:07.676532 17612 ts_tablet_manager.cc:1939] T 568ac290961248cab8904dcdea6c6167 P 3c4cb3d7c2c749558a80ee3ad051c341: Deleting consensus metadata
I20260926 01:02:07.676939 17092 catalog_manager.cc:4841] TS 3c4cb3d7c2c749558a80ee3ad051c341 (127.3.75.2:34311): tablet 568ac290961248cab8904dcdea6c6167 (table test-table [id=2409715a68f749fcbbfe088b0fd29a88]) successfully deleted
W20260926 01:02:07.679836 17091 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.75.1:43553 (error 108)
I20260926 01:02:07.680086 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 17305
I20260926 01:02:07.680253 17585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:07.680144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.1:43553 (local address 127.0.0.1:58574)
0926 01:02:07.680183 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:07.680216 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.1:43553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260926 01:02:07.681674 17091 catalog_manager.cc:4584] TS ceb466341f8246e990249e49378d40ba (127.3.75.1:43553): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 568ac290961248cab8904dcdea6c6167: Network error: Client connection negotiation failed: client connection to 127.3.75.1:43553: connect: Connection refused (error 111)
I20260926 01:02:07.684971 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 17440
I20260926 01:02:07.690264 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 17075
I20260926 01:02:07.695228 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:34273
--webserver_interface=127.3.75.62
--webserver_port=37489
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:37523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.75.62:34273
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:02:07.777428 17614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:02:07.777700 17614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:02:07.777747 17614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:02:07.779081 17614 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:02:07.779166 17614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:02:07.779196 17614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:02:07.779222 17614 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:02:07.779253 17614 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:02:07.780920 17614 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:37523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.3.75.62:34273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:34273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.75.62
--webserver_port=37489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:02:07.781234 17614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:02:07.781492 17614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:02:07.790613 17621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:02:07.790685 17614 server_base.cc:1034] running on GCE node
W20260926 01:02:07.790807 17623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:02:07.790670 17620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:02:07.791185 17614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:02:07.791491 17614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:02:07.792629 17614 hybrid_clock.cc:648] HybridClock initialized: now 1790384527792606 us; error 42 us; skew 500 ppm
Sep 26 01:02:07 dist-test-slave-7v10 krb5kdc[17062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384527, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:02:07.795109 17614 init.cc:377] Logged in from keytab as kudu/127.3.75.62@KRBTEST.COM (short username kudu)
I20260926 01:02:07.795512 17614 webserver.cc:460] Webserver started at http://127.3.75.62:37489/ using document root <none> and password file <none>
I20260926 01:02:07.795753 17614 fs_manager.cc:359] Metadata directory not provided
I20260926 01:02:07.795830 17614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:02:07.797153 17614 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:02:07.797778 17630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:02:07.797964 17614 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 01:02:07.798072 17614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/master-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/master-0/wal
uuid: "aff9338d776a42d18e1ae239f17fa778"
format_stamp: "Formatted at 2026-09-26 01:02:07 on dist-test-slave-7v10"
server_key: "560ae9ec39a670f2ef4d5ee4c143b20e"
server_key_iv: "880158daa51a878aaa41ad79645afad3"
server_key_version: "encryptionkey@0"
I20260926 01:02:07.798475 17614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:02:07.819594 17614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:02:07.821430 17614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:02:07.821595 17614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:02:07.825485 17614 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.62:34273
I20260926 01:02:07.825523 17683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.62:34273 every 8 connection(s)
I20260926 01:02:07.825911 17614 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
I20260926 01:02:07.827476 17684 sys_catalog.cc:263] Verifying existing consensus state
I20260926 01:02:07.828130 17684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778: Bootstrap starting.
I20260926 01:02:07.830143 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 17614
I20260926 01:02:07.830462 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.1:43553
--local_ip_for_outbound_sockets=127.3.75.1
--tserver_master_addrs=127.3.75.62:34273
--webserver_port=36707
--webserver_interface=127.3.75.1
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab
--principal=kudu/127.3.75.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:37523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 01:02:07.836818 17684 log.cc:824] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778: Log is configured to *not* fsync() on all Append() calls
I20260926 01:02:07.838735 17684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 01:02:07.839069 17684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778: Bootstrap complete.
I20260926 01:02:07.841394 17684 raft_consensus.cc:348] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff9338d776a42d18e1ae239f17fa778" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 34273 } }
I20260926 01:02:07.841642 17684 raft_consensus.cc:729] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aff9338d776a42d18e1ae239f17fa778, State: Initialized, Role: FOLLOWER
I20260926 01:02:07.841845 17684 consensus_queue.cc:260] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff9338d776a42d18e1ae239f17fa778" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 34273 } }
I20260926 01:02:07.841941 17684 raft_consensus.cc:388] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:02:07.842000 17684 raft_consensus.cc:482] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:02:07.842049 17684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778 [term 1 FOLLOWER]: Advancing to term 2
I20260926 01:02:07.843094 17684 raft_consensus.cc:504] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff9338d776a42d18e1ae239f17fa778" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 34273 } }
I20260926 01:02:07.843212 17684 leader_election.cc:302] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aff9338d776a42d18e1ae239f17fa778; no voters:
I20260926 01:02:07.843405 17684 leader_election.cc:288] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 01:02:07.843439 17688 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 01:02:07.843549 17688 raft_consensus.cc:686] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778 [term 2 LEADER]: Becoming Leader. State: Replica: aff9338d776a42d18e1ae239f17fa778, State: Running, Role: LEADER
I20260926 01:02:07.843675 17684 sys_catalog.cc:564] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:02:07.843709 17688 consensus_queue.cc:237] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff9338d776a42d18e1ae239f17fa778" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 34273 } }
I20260926 01:02:07.844141 17690 sys_catalog.cc:455] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aff9338d776a42d18e1ae239f17fa778. Latest consensus state: current_term: 2 leader_uuid: "aff9338d776a42d18e1ae239f17fa778" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff9338d776a42d18e1ae239f17fa778" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 34273 } } }
I20260926 01:02:07.844153 17689 sys_catalog.cc:455] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "aff9338d776a42d18e1ae239f17fa778" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff9338d776a42d18e1ae239f17fa778" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 34273 } } }
I20260926 01:02:07.844234 17690 sys_catalog.cc:458] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778 [sys.catalog]: This master's current role is: LEADER
I20260926 01:02:07.844261 17689 sys_catalog.cc:458] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778 [sys.catalog]: This master's current role is: LEADER
I20260926 01:02:07.844473 17695 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:02:07.845345 17695 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:02:07.845489 17695 catalog_manager.cc:1259] Loaded cluster ID: aef9d48a0bd14f528e23b4429a7e4d14
I20260926 01:02:07.845754 17695 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:02:07.846071 17695 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260926 01:02:07.851210 17695 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260926 01:02:07.853654 17695 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:02:07.853876 17695 catalog_manager.cc:5879] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778: Loaded TSK: 0
I20260926 01:02:07.854183 17695 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 01:02:07.914099 17686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:02:07.914356 17686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:02:07.914397 17686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:02:07.916172 17686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:02:07.916245 17686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:02:07.916294 17686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.1
I20260926 01:02:07.917810 17686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:37523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab
--principal=kudu/127.3.75.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.1:43553
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.75.1
--webserver_port=36707
--tserver_master_addrs=127.3.75.62:34273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.1
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:02:07.918076 17686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:02:07.918310 17686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:02:07.924351 17712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:02:07.924360 17711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:02:07.924504 17686 server_base.cc:1034] running on GCE node
W20260926 01:02:07.924613 17714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:02:07.924831 17686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:02:07.925130 17686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:02:07.926316 17686 hybrid_clock.cc:648] HybridClock initialized: now 1790384527926295 us; error 32 us; skew 500 ppm
Sep 26 01:02:07 dist-test-slave-7v10 krb5kdc[17062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384527, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:02:07.937598 17686 init.cc:377] Logged in from keytab as kudu/127.3.75.1@KRBTEST.COM (short username kudu)
I20260926 01:02:07.938095 17686 webserver.cc:460] Webserver started at http://127.3.75.1:36707/ using document root <none> and password file <none>
I20260926 01:02:07.938365 17686 fs_manager.cc:359] Metadata directory not provided
I20260926 01:02:07.938422 17686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:02:07.939888 17686 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:02:07.940588 17721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:02:07.940806 17686 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:02:07.940932 17686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-0/wal
uuid: "ceb466341f8246e990249e49378d40ba"
format_stamp: "Formatted at 2026-09-26 01:02:07 on dist-test-slave-7v10"
server_key: "ad543bae962dbd335f350cfbe0aa0e1b"
server_key_iv: "4545a50ad287114ad5d185b87cc89683"
server_key_version: "encryptionkey@0"
I20260926 01:02:07.941282 17686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:02:07.950555 17686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:02:07.952867 17686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:02:07.953023 17686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:02:07.953300 17686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:02:07.954525 17729 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 01:02:07.956053 17686 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 01:02:07.956126 17686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 01:02:07.956171 17686 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 01:02:07.956813 17686 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 01:02:07.956858 17686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 01:02:07.956918 17729 tablet_bootstrap.cc:492] T 568ac290961248cab8904dcdea6c6167 P ceb466341f8246e990249e49378d40ba: Bootstrap starting.
I20260926 01:02:07.964671 17686 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.1:43553
I20260926 01:02:07.964730 17836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.1:43553 every 8 connection(s)
I20260926 01:02:07.965255 17686 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
I20260926 01:02:07.965742 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 17686
I20260926 01:02:07.966022 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.2:34311
--local_ip_for_outbound_sockets=127.3.75.2
--tserver_master_addrs=127.3.75.62:34273
--webserver_port=36831
--webserver_interface=127.3.75.2
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab
--principal=kudu/127.3.75.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:37523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 01:02:07.968452 17729 log.cc:824] T 568ac290961248cab8904dcdea6c6167 P ceb466341f8246e990249e49378d40ba: Log is configured to *not* fsync() on all Append() calls
I20260926 01:02:07.971463 17729 tablet_bootstrap.cc:492] T 568ac290961248cab8904dcdea6c6167 P ceb466341f8246e990249e49378d40ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 01:02:07.971803 17729 tablet_bootstrap.cc:492] T 568ac290961248cab8904dcdea6c6167 P ceb466341f8246e990249e49378d40ba: Bootstrap complete.
I20260926 01:02:07.973074 17729 ts_tablet_manager.cc:1397] T 568ac290961248cab8904dcdea6c6167 P ceb466341f8246e990249e49378d40ba: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.011s
I20260926 01:02:07.974835 17729 raft_consensus.cc:348] T 568ac290961248cab8904dcdea6c6167 P ceb466341f8246e990249e49378d40ba [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4cb3d7c2c749558a80ee3ad051c341" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 34311 } } peers { permanent_uuid: "ceb466341f8246e990249e49378d40ba" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 43553 } } peers { permanent_uuid: "075f2a85cca14f3c892ecaeba6244ea4" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 39451 } }
I20260926 01:02:07.975250 17729 raft_consensus.cc:729] T 568ac290961248cab8904dcdea6c6167 P ceb466341f8246e990249e49378d40ba [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceb466341f8246e990249e49378d40ba, State: Initialized, Role: FOLLOWER
I20260926 01:02:07.975432 17729 consensus_queue.cc:260] T 568ac290961248cab8904dcdea6c6167 P ceb466341f8246e990249e49378d40ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4cb3d7c2c749558a80ee3ad051c341" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 34311 } } peers { permanent_uuid: "ceb466341f8246e990249e49378d40ba" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 43553 } } peers { permanent_uuid: "075f2a85cca14f3c892ecaeba6244ea4" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 39451 } }
I20260926 01:02:07.975939 17729 ts_tablet_manager.cc:1428] T 568ac290961248cab8904dcdea6c6167 P ceb466341f8246e990249e49378d40ba: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
Sep 26 01:02:07 dist-test-slave-7v10 krb5kdc[17062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384527, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.1@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 01:02:07.981699 17841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:07.967991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:51911 (local address 127.3.75.62:34273)
0926 01:02:07.968776 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:07.968796 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:07.972014 (+ 3218us) server_negotiation.cc:407] Connection header received
0926 01:02:07.972667 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:07.972673 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:07.972715 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:07.972870 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:02:07.973554 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:07.974412 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:07.975602 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:07.976028 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:07.976130 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:07.978335 (+ 2205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:02:07.978338 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:02:07.978339 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:02:07.978352 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:02:07.979944 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:07.980447 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:07.980450 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:07.980450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:07.980541 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:07.980803 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:07.980804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:07.980805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:07.981015 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:02:07.981104 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:07.981400 (+ 296us) server_negotiation.cc:299] Negotiation successful
0926 01:02:07.981522 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":727,"thread_start_us":86,"threads_started":1}
I20260926 01:02:07.981711 17840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:07.967941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34273 (local address 127.3.75.1:51911)
0926 01:02:07.968877 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:07.968921 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 01:02:07.972435 (+ 3514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:02:07.972902 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:02:07.972906 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:02:07.973235 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:02:07.973439 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:07.973445 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:07.974529 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:07.974532 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:07.975492 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:07.975494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:07.976205 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:07.976206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:07.976300 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:07.976922 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:02:07.976930 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:02:07.978229 (+ 1299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:02:07.980045 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:02:07.980052 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:02:07.980055 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:02:07.980363 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:02:07.980616 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:02:07.980617 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:02:07.980617 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:02:07.980728 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:02:07.981095 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:02:07.981102 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:02:07.981235 (+ 133us) client_negotiation.cc:769] Sending connection context
0926 01:02:07.981336 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 01:02:07.981505 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":894,"thread_start_us":100,"threads_started":1}
I20260926 01:02:07.982343 17837 heartbeater.cc:347] Connected to a master server at 127.3.75.62:34273
I20260926 01:02:07.982431 17837 heartbeater.cc:464] Registering TS with master...
I20260926 01:02:07.982628 17837 heartbeater.cc:511] Master 127.3.75.62:34273 requested a full tablet report, sending...
I20260926 01:02:07.983336 17648 ts_manager.cc:194] Registered new tserver with Master: ceb466341f8246e990249e49378d40ba (127.3.75.1:43553)
I20260926 01:02:07.984385 17648 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.75.1@KRBTEST.COM'} at 127.3.75.1:51911
I20260926 01:02:07.985334 17837 heartbeater.cc:503] Master 127.3.75.62:34273 was elected leader, sending a full tablet report...
Sep 26 01:02:07 dist-test-slave-7v10 krb5kdc[17062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384527, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.62@KRBTEST.COM for kudu/127.3.75.1@KRBTEST.COM
I20260926 01:02:07.992113 17844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:07.983881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.1:43553 (local address 127.0.0.1:58576)
0926 01:02:07.984169 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:07.984210 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 01:02:07.984286 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:02:07.984471 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:02:07.984476 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:02:07.984780 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:02:07.984840 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:07.984842 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:07.986048 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:07.986049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:07.986957 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:07.986960 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:07.987430 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:07.987433 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:07.987509 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:07.987947 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:02:07.987952 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:02:07.989292 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:02:07.991132 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:02:07.991138 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:02:07.991139 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:02:07.991324 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:02:07.991532 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:02:07.991532 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:02:07.991532 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:02:07.991583 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:02:07.991867 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:02:07.991872 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:02:07.991910 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 01:02:07.991969 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 01:02:07.992048 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":82,"threads_started":1}
I20260926 01:02:07.992210 17845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:07.983953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58576 (local address 127.3.75.1:43553)
0926 01:02:07.984101 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:07.984105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:07.984294 (+ 189us) server_negotiation.cc:407] Connection header received
0926 01:02:07.984362 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:07.984364 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:07.984379 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:07.984456 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:02:07.984915 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:07.985965 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:07.987035 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:07.987352 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:07.987405 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:07.989371 (+ 1966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:02:07.989379 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:02:07.989379 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:02:07.989386 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:02:07.991059 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:07.991401 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:07.991404 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:07.991404 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:07.991467 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:07.991641 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:07.991642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:07.991642 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:07.991806 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:02:07.991885 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:07.992026 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 01:02:07.992115 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":122,"thread_start_us":78,"threads_started":1}
I20260926 01:02:07.992483 17759 tablet_service.cc:1514] Processing DeleteTablet for tablet 568ac290961248cab8904dcdea6c6167 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:02:07 UTC) from {username='kudu', principal='kudu/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:58576
I20260926 01:02:07.993117 17846 tablet_replica.cc:330] stopping tablet replica
I20260926 01:02:07.993221 17846 raft_consensus.cc:2229] T 568ac290961248cab8904dcdea6c6167 P ceb466341f8246e990249e49378d40ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:02:07.993289 17846 raft_consensus.cc:2258] T 568ac290961248cab8904dcdea6c6167 P ceb466341f8246e990249e49378d40ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:02:07.993568 17846 ts_tablet_manager.cc:1905] T 568ac290961248cab8904dcdea6c6167 P ceb466341f8246e990249e49378d40ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:02:07.995452 17846 ts_tablet_manager.cc:1918] T 568ac290961248cab8904dcdea6c6167 P ceb466341f8246e990249e49378d40ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:02:07.995524 17846 log.cc:1196] T 568ac290961248cab8904dcdea6c6167 P ceb466341f8246e990249e49378d40ba: Deleting WAL directory at /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-0/wal/wals/568ac290961248cab8904dcdea6c6167
I20260926 01:02:07.995795 17846 ts_tablet_manager.cc:1939] T 568ac290961248cab8904dcdea6c6167 P ceb466341f8246e990249e49378d40ba: Deleting consensus metadata
I20260926 01:02:07.996263 17631 catalog_manager.cc:4841] TS ceb466341f8246e990249e49378d40ba (127.3.75.1:43553): tablet 568ac290961248cab8904dcdea6c6167 (table test-table [id=2409715a68f749fcbbfe088b0fd29a88]) successfully deleted
W20260926 01:02:08.051607 17839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:02:08.051828 17839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:02:08.051865 17839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:02:08.053050 17839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:02:08.053119 17839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:02:08.053162 17839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.2
I20260926 01:02:08.054575 17839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:37523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab
--principal=kudu/127.3.75.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.2:34311
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.75.2
--webserver_port=36831
--tserver_master_addrs=127.3.75.62:34273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.2
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:02:08.054829 17839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:02:08.055032 17839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:02:08.059585 17851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:02:08.059593 17852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:02:08.059603 17854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:02:08.059674 17839 server_base.cc:1034] running on GCE node
I20260926 01:02:08.059996 17839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:02:08.060242 17839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:02:08.061406 17839 hybrid_clock.cc:648] HybridClock initialized: now 1790384528061391 us; error 35 us; skew 500 ppm
Sep 26 01:02:08 dist-test-slave-7v10 krb5kdc[17062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384528, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:02:08.063745 17839 init.cc:377] Logged in from keytab as kudu/127.3.75.2@KRBTEST.COM (short username kudu)
I20260926 01:02:08.064149 17839 webserver.cc:460] Webserver started at http://127.3.75.2:36831/ using document root <none> and password file <none>
I20260926 01:02:08.064383 17839 fs_manager.cc:359] Metadata directory not provided
I20260926 01:02:08.064462 17839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:02:08.065711 17839 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:02:08.066224 17861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:02:08.066365 17839 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:02:08.066457 17839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-1/wal
uuid: "3c4cb3d7c2c749558a80ee3ad051c341"
format_stamp: "Formatted at 2026-09-26 01:02:07 on dist-test-slave-7v10"
server_key: "341c439563262cc3d449fe7d9c373127"
server_key_iv: "1855dd36ad1a53a271802b4076d2b444"
server_key_version: "encryptionkey@0"
I20260926 01:02:08.066747 17839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:02:08.079610 17839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:02:08.081370 17839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:02:08.081518 17839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:02:08.081722 17839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:02:08.082005 17839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:02:08.082056 17839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:02:08.082095 17839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:02:08.082124 17839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:02:08.087922 17839 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.2:34311
I20260926 01:02:08.087963 17975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.2:34311 every 8 connection(s)
I20260926 01:02:08.088294 17839 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
I20260926 01:02:08.091926 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 17839
I20260926 01:02:08.092206 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.3:39451
--local_ip_for_outbound_sockets=127.3.75.3
--tserver_master_addrs=127.3.75.62:34273
--webserver_port=32939
--webserver_interface=127.3.75.3
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab
--principal=kudu/127.3.75.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:37523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 01:02:08 dist-test-slave-7v10 krb5kdc[17062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384528, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.2@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 01:02:08.098223 17978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:08.088998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34273 (local address 127.3.75.2:37673)
0926 01:02:08.089160 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:08.089196 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 01:02:08.089661 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:02:08.089858 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:02:08.089861 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:02:08.090414 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:02:08.090603 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:08.090608 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:08.091290 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:08.091290 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:08.092378 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:08.092381 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:08.092735 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:08.092736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:08.092899 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:08.093420 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:02:08.093429 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:02:08.095067 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:02:08.096940 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:02:08.096945 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:02:08.096947 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:02:08.097158 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:02:08.097374 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:02:08.097374 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:02:08.097374 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:02:08.097466 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:02:08.097758 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:02:08.097762 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:02:08.097873 (+ 111us) client_negotiation.cc:769] Sending connection context
0926 01:02:08.097951 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 01:02:08.098066 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":86,"threads_started":1}
I20260926 01:02:08.098238 17841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:08.088997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:37673 (local address 127.3.75.62:34273)
0926 01:02:08.089044 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:08.089047 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:08.089282 (+ 235us) server_negotiation.cc:407] Connection header received
0926 01:02:08.089739 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:08.089740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:08.089753 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:08.089807 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:02:08.090677 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:08.091213 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:08.092458 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:08.092652 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:08.092705 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:08.095152 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:02:08.095154 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:02:08.095154 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:02:08.095162 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:02:08.096834 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:08.097231 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:08.097232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:08.097233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:08.097304 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:08.097532 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:08.097533 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:08.097533 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:08.097688 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:02:08.097765 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:08.098000 (+ 235us) server_negotiation.cc:299] Negotiation successful
0926 01:02:08.098098 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 01:02:08.098860 17976 heartbeater.cc:347] Connected to a master server at 127.3.75.62:34273
I20260926 01:02:08.099015 17976 heartbeater.cc:464] Registering TS with master...
I20260926 01:02:08.099187 17976 heartbeater.cc:511] Master 127.3.75.62:34273 requested a full tablet report, sending...
I20260926 01:02:08.099627 17648 ts_manager.cc:194] Registered new tserver with Master: 3c4cb3d7c2c749558a80ee3ad051c341 (127.3.75.2:34311)
I20260926 01:02:08.100171 17648 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.75.2@KRBTEST.COM'} at 127.3.75.2:37673
W20260926 01:02:08.173162 17979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:02:08.173357 17979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:02:08.173379 17979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:02:08.174438 17979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:02:08.174469 17979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:02:08.174494 17979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.3
I20260926 01:02:08.175817 17979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:37523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab
--principal=kudu/127.3.75.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.3:39451
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.75.3
--webserver_port=32939
--tserver_master_addrs=127.3.75.62:34273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.3
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:02:08.175978 17979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:02:08.176136 17979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:02:08.180819 17987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:02:08.180833 17984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:02:08.180826 17985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:02:08.181032 17979 server_base.cc:1034] running on GCE node
I20260926 01:02:08.181191 17979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:02:08.181367 17979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:02:08.182528 17979 hybrid_clock.cc:648] HybridClock initialized: now 1790384528182513 us; error 25 us; skew 500 ppm
Sep 26 01:02:08 dist-test-slave-7v10 krb5kdc[17062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384528, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:02:08.184890 17979 init.cc:377] Logged in from keytab as kudu/127.3.75.3@KRBTEST.COM (short username kudu)
I20260926 01:02:08.185318 17979 webserver.cc:460] Webserver started at http://127.3.75.3:32939/ using document root <none> and password file <none>
I20260926 01:02:08.185528 17979 fs_manager.cc:359] Metadata directory not provided
I20260926 01:02:08.185621 17979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:02:08.186859 17979 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:02:08.187508 17994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:02:08.187665 17979 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 01:02:08.187762 17979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-2/wal
uuid: "075f2a85cca14f3c892ecaeba6244ea4"
format_stamp: "Formatted at 2026-09-26 01:02:07 on dist-test-slave-7v10"
server_key: "82af5ba29d9171d4f288cc330359ee73"
server_key_iv: "3dd837ed4adcb4be2e765588e539c9f3"
server_key_version: "encryptionkey@0"
I20260926 01:02:08.188061 17979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:02:08.201742 17979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:02:08.203648 17979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:02:08.203802 17979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:02:08.204015 17979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:02:08.204298 17979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:02:08.204346 17979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:02:08.204386 17979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:02:08.204414 17979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:02:08.210237 17979 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.3:39451
I20260926 01:02:08.210296 18108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.3:39451 every 8 connection(s)
I20260926 01:02:08.210587 17979 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
Sep 26 01:02:08 dist-test-slave-7v10 krb5kdc[17062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384528, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.3@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 01:02:08.217609 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 17979
I20260926 01:02:08.219213 17841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:08.211299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.3:52805 (local address 127.3.75.62:34273)
0926 01:02:08.211344 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:08.211346 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:08.211542 (+ 196us) server_negotiation.cc:407] Connection header received
0926 01:02:08.211980 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:08.211981 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:08.211990 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:08.212045 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:02:08.212876 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:08.213363 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:08.214229 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:08.214421 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:08.214474 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:08.216393 (+ 1919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:02:08.216394 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:02:08.216394 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:02:08.216401 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:02:08.217812 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:08.218277 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:08.218278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:08.218279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:08.218343 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:08.218569 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:08.218570 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:08.218570 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:08.218713 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:02:08.218791 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:08.219074 (+ 283us) server_negotiation.cc:299] Negotiation successful
0926 01:02:08.219149 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 01:02:08.219455 18111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:08.211266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34273 (local address 127.3.75.3:52805)
0926 01:02:08.211413 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:08.211452 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 01:02:08.211902 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:02:08.212094 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:02:08.212097 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:02:08.212620 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:02:08.212806 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:08.212810 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:08.213437 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:08.213438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:08.214169 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:08.214171 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:08.214500 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:08.214501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:08.214600 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:08.215034 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:02:08.215041 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:02:08.216308 (+ 1267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:02:08.217881 (+ 1573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:02:08.217887 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:02:08.217889 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:02:08.218187 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:02:08.218408 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:02:08.218409 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:02:08.218409 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:02:08.218503 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:02:08.218786 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:02:08.218790 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:02:08.218950 (+ 160us) client_negotiation.cc:769] Sending connection context
0926 01:02:08.219030 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 01:02:08.219203 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":82,"threads_started":1}
I20260926 01:02:08.219866 18109 heartbeater.cc:347] Connected to a master server at 127.3.75.62:34273
I20260926 01:02:08.219985 18109 heartbeater.cc:464] Registering TS with master...
I20260926 01:02:08.220162 18109 heartbeater.cc:511] Master 127.3.75.62:34273 requested a full tablet report, sending...
I20260926 01:02:08.220551 17648 ts_manager.cc:194] Registered new tserver with Master: 075f2a85cca14f3c892ecaeba6244ea4 (127.3.75.3:39451)
I20260926 01:02:08.221025 17648 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.75.3@KRBTEST.COM'} at 127.3.75.3:52805
I20260926 01:02:08.224412 18113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:08.217975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33066 (local address 127.3.75.62:34273)
0926 01:02:08.218109 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:08.218111 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:08.218156 (+ 45us) server_negotiation.cc:407] Connection header received
0926 01:02:08.218205 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:08.218206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:08.218218 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:08.218285 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:02:08.219073 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:08.219836 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:08.220835 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:08.221148 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:08.221228 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:08.222394 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:02:08.222395 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:02:08.222395 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:02:08.222400 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:02:08.223568 (+ 1168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:08.223857 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:08.223858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:08.223858 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:08.223904 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:08.224042 (+ 138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:08.224042 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:08.224042 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:08.224120 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:02:08.224220 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:08.224271 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 01:02:08.224327 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":81,"threads_started":1}
I20260926 01:02:08.225025 3372 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 01:02:08.227380 18113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:08.225325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33078 (local address 127.3.75.62:34273)
0926 01:02:08.225390 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:08.225392 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:08.225454 (+ 62us) server_negotiation.cc:407] Connection header received
0926 01:02:08.225516 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:08.225516 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:08.225540 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:08.225600 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:02:08.225668 (+ 68us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:08.226085 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:08.226738 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:08.226971 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:08.227021 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:08.227141 (+ 120us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:02:08.227222 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:02:08.227284 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:08.227320 (+ 36us) server_negotiation.cc:299] Negotiation successful
0926 01:02:08.227336 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 01:02:08.227643 17648 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:33078:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:02:08.229247 17648 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:02:08.231544 17759 tablet_service.cc:1467] Processing CreateTablet for tablet 60612f58ad1c4ed3bc8f2f460282f4ad (DEFAULT_TABLE table=test-table [id=85299fabc3284272929e1fc07125bc33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:02:08.231786 17759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60612f58ad1c4ed3bc8f2f460282f4ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:02:08.233413 17729 tablet_bootstrap.cc:492] T 60612f58ad1c4ed3bc8f2f460282f4ad P ceb466341f8246e990249e49378d40ba: Bootstrap starting.
I20260926 01:02:08.233842 17729 tablet_bootstrap.cc:654] T 60612f58ad1c4ed3bc8f2f460282f4ad P ceb466341f8246e990249e49378d40ba: Neither blocks nor log segments found. Creating new log.
I20260926 01:02:08.234772 17729 tablet_bootstrap.cc:492] T 60612f58ad1c4ed3bc8f2f460282f4ad P ceb466341f8246e990249e49378d40ba: No bootstrap required, opened a new log
I20260926 01:02:08.234838 17729 ts_tablet_manager.cc:1397] T 60612f58ad1c4ed3bc8f2f460282f4ad P ceb466341f8246e990249e49378d40ba: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 01:02:08.235033 17729 raft_consensus.cc:348] T 60612f58ad1c4ed3bc8f2f460282f4ad P ceb466341f8246e990249e49378d40ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4cb3d7c2c749558a80ee3ad051c341" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 34311 } } peers { permanent_uuid: "075f2a85cca14f3c892ecaeba6244ea4" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 39451 } } peers { permanent_uuid: "ceb466341f8246e990249e49378d40ba" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 43553 } }
I20260926 01:02:08.235116 17729 raft_consensus.cc:374] T 60612f58ad1c4ed3bc8f2f460282f4ad P ceb466341f8246e990249e49378d40ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:02:08.235168 17729 raft_consensus.cc:729] T 60612f58ad1c4ed3bc8f2f460282f4ad P ceb466341f8246e990249e49378d40ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceb466341f8246e990249e49378d40ba, State: Initialized, Role: FOLLOWER
I20260926 01:02:08.235215 17729 consensus_queue.cc:260] T 60612f58ad1c4ed3bc8f2f460282f4ad P ceb466341f8246e990249e49378d40ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4cb3d7c2c749558a80ee3ad051c341" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 34311 } } peers { permanent_uuid: "075f2a85cca14f3c892ecaeba6244ea4" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 39451 } } peers { permanent_uuid: "ceb466341f8246e990249e49378d40ba" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 43553 } }
I20260926 01:02:08.235330 17729 ts_tablet_manager.cc:1428] T 60612f58ad1c4ed3bc8f2f460282f4ad P ceb466341f8246e990249e49378d40ba: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 01:02:08.235589 18115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:08.231108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.3:39451 (local address 127.0.0.1:50946)
0926 01:02:08.231234 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:08.231244 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:02:08.231290 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:02:08.231778 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:02:08.231779 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:02:08.231781 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:02:08.231824 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:08.231826 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:08.233060 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:08.233061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:08.234119 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:08.234123 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:08.235338 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:08.235339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:08.235399 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:08.235403 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:02:08.235461 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 01:02:08.235509 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
I20260926 01:02:08.235599 18117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:08.231385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50946 (local address 127.3.75.3:39451)
0926 01:02:08.231571 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:08.231575 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:08.231584 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:02:08.231618 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:08.231619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:08.231632 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:08.231701 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:02:08.231907 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:08.232967 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:08.234478 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:08.235188 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:08.235249 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:08.235302 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:08.235480 (+ 178us) server_negotiation.cc:299] Negotiation successful
0926 01:02:08.235518 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":90,"threads_started":1}
I20260926 01:02:08.235670 17844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:08.231058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.2:34311 (local address 127.0.0.1:35882)
0926 01:02:08.231091 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:08.231101 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:02:08.231140 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:02:08.231653 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:02:08.231653 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:02:08.231655 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:02:08.231697 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:08.231699 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:08.233061 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:08.233062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:08.234075 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:08.234078 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:08.235268 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:08.235270 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:08.235342 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:08.235366 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 01:02:08.235428 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 01:02:08.235479 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17,"mutex_wait_us":41}
I20260926 01:02:08.235935 18116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:08.231163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35882 (local address 127.3.75.2:34311)
0926 01:02:08.231452 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:08.231455 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:08.231463 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:02:08.231520 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:08.231522 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:08.231538 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:08.231630 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:02:08.231804 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:08.232915 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:08.234249 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:08.235058 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:08.235153 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:08.235217 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:08.235716 (+ 499us) server_negotiation.cc:299] Negotiation successful
0926 01:02:08.235762 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":137,"threads_started":1}
I20260926 01:02:08.236450 18043 tablet_service.cc:1467] Processing CreateTablet for tablet 60612f58ad1c4ed3bc8f2f460282f4ad (DEFAULT_TABLE table=test-table [id=85299fabc3284272929e1fc07125bc33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:02:08.236572 17910 tablet_service.cc:1467] Processing CreateTablet for tablet 60612f58ad1c4ed3bc8f2f460282f4ad (DEFAULT_TABLE table=test-table [id=85299fabc3284272929e1fc07125bc33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:02:08.236827 18043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60612f58ad1c4ed3bc8f2f460282f4ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:02:08.236940 17910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60612f58ad1c4ed3bc8f2f460282f4ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:02:08.239660 18120 tablet_bootstrap.cc:492] T 60612f58ad1c4ed3bc8f2f460282f4ad P 075f2a85cca14f3c892ecaeba6244ea4: Bootstrap starting.
I20260926 01:02:08.239650 18119 tablet_bootstrap.cc:492] T 60612f58ad1c4ed3bc8f2f460282f4ad P 3c4cb3d7c2c749558a80ee3ad051c341: Bootstrap starting.
I20260926 01:02:08.240276 18119 tablet_bootstrap.cc:654] T 60612f58ad1c4ed3bc8f2f460282f4ad P 3c4cb3d7c2c749558a80ee3ad051c341: Neither blocks nor log segments found. Creating new log.
I20260926 01:02:08.240279 18120 tablet_bootstrap.cc:654] T 60612f58ad1c4ed3bc8f2f460282f4ad P 075f2a85cca14f3c892ecaeba6244ea4: Neither blocks nor log segments found. Creating new log.
I20260926 01:02:08.240787 18120 log.cc:824] T 60612f58ad1c4ed3bc8f2f460282f4ad P 075f2a85cca14f3c892ecaeba6244ea4: Log is configured to *not* fsync() on all Append() calls
I20260926 01:02:08.240795 18119 log.cc:824] T 60612f58ad1c4ed3bc8f2f460282f4ad P 3c4cb3d7c2c749558a80ee3ad051c341: Log is configured to *not* fsync() on all Append() calls
I20260926 01:02:08.241515 18120 tablet_bootstrap.cc:492] T 60612f58ad1c4ed3bc8f2f460282f4ad P 075f2a85cca14f3c892ecaeba6244ea4: No bootstrap required, opened a new log
I20260926 01:02:08.241515 18119 tablet_bootstrap.cc:492] T 60612f58ad1c4ed3bc8f2f460282f4ad P 3c4cb3d7c2c749558a80ee3ad051c341: No bootstrap required, opened a new log
I20260926 01:02:08.241569 18120 ts_tablet_manager.cc:1397] T 60612f58ad1c4ed3bc8f2f460282f4ad P 075f2a85cca14f3c892ecaeba6244ea4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:02:08.241595 18119 ts_tablet_manager.cc:1397] T 60612f58ad1c4ed3bc8f2f460282f4ad P 3c4cb3d7c2c749558a80ee3ad051c341: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:02:08.243046 18120 raft_consensus.cc:348] T 60612f58ad1c4ed3bc8f2f460282f4ad P 075f2a85cca14f3c892ecaeba6244ea4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4cb3d7c2c749558a80ee3ad051c341" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 34311 } } peers { permanent_uuid: "075f2a85cca14f3c892ecaeba6244ea4" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 39451 } } peers { permanent_uuid: "ceb466341f8246e990249e49378d40ba" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 43553 } }
I20260926 01:02:08.243228 18120 raft_consensus.cc:374] T 60612f58ad1c4ed3bc8f2f460282f4ad P 075f2a85cca14f3c892ecaeba6244ea4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:02:08.243036 18119 raft_consensus.cc:348] T 60612f58ad1c4ed3bc8f2f460282f4ad P 3c4cb3d7c2c749558a80ee3ad051c341 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4cb3d7c2c749558a80ee3ad051c341" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 34311 } } peers { permanent_uuid: "075f2a85cca14f3c892ecaeba6244ea4" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 39451 } } peers { permanent_uuid: "ceb466341f8246e990249e49378d40ba" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 43553 } }
I20260926 01:02:08.243258 18120 raft_consensus.cc:729] T 60612f58ad1c4ed3bc8f2f460282f4ad P 075f2a85cca14f3c892ecaeba6244ea4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075f2a85cca14f3c892ecaeba6244ea4, State: Initialized, Role: FOLLOWER
I20260926 01:02:08.243289 18119 raft_consensus.cc:374] T 60612f58ad1c4ed3bc8f2f460282f4ad P 3c4cb3d7c2c749558a80ee3ad051c341 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:02:08.243338 18119 raft_consensus.cc:729] T 60612f58ad1c4ed3bc8f2f460282f4ad P 3c4cb3d7c2c749558a80ee3ad051c341 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4cb3d7c2c749558a80ee3ad051c341, State: Initialized, Role: FOLLOWER
I20260926 01:02:08.243396 18120 consensus_queue.cc:260] T 60612f58ad1c4ed3bc8f2f460282f4ad P 075f2a85cca14f3c892ecaeba6244ea4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4cb3d7c2c749558a80ee3ad051c341" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 34311 } } peers { permanent_uuid: "075f2a85cca14f3c892ecaeba6244ea4" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 39451 } } peers { permanent_uuid: "ceb466341f8246e990249e49378d40ba" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 43553 } }
I20260926 01:02:08.243469 18119 consensus_queue.cc:260] T 60612f58ad1c4ed3bc8f2f460282f4ad P 3c4cb3d7c2c749558a80ee3ad051c341 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4cb3d7c2c749558a80ee3ad051c341" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 34311 } } peers { permanent_uuid: "075f2a85cca14f3c892ecaeba6244ea4" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 39451 } } peers { permanent_uuid: "ceb466341f8246e990249e49378d40ba" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 43553 } }
I20260926 01:02:08.243726 18120 ts_tablet_manager.cc:1428] T 60612f58ad1c4ed3bc8f2f460282f4ad P 075f2a85cca14f3c892ecaeba6244ea4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 01:02:08.243760 18109 heartbeater.cc:503] Master 127.3.75.62:34273 was elected leader, sending a full tablet report...
I20260926 01:02:08.244029 17976 heartbeater.cc:503] Master 127.3.75.62:34273 was elected leader, sending a full tablet report...
I20260926 01:02:08.244330 18119 ts_tablet_manager.cc:1428] T 60612f58ad1c4ed3bc8f2f460282f4ad P 3c4cb3d7c2c749558a80ee3ad051c341: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 01:02:08.292593 18124 raft_consensus.cc:482] T 60612f58ad1c4ed3bc8f2f460282f4ad P 3c4cb3d7c2c749558a80ee3ad051c341 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:02:08.292775 18124 raft_consensus.cc:504] T 60612f58ad1c4ed3bc8f2f460282f4ad P 3c4cb3d7c2c749558a80ee3ad051c341 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4cb3d7c2c749558a80ee3ad051c341" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 34311 } } peers { permanent_uuid: "075f2a85cca14f3c892ecaeba6244ea4" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 39451 } } peers { permanent_uuid: "ceb466341f8246e990249e49378d40ba" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 43553 } }
I20260926 01:02:08.293145 18124 leader_election.cc:288] T 60612f58ad1c4ed3bc8f2f460282f4ad P 3c4cb3d7c2c749558a80ee3ad051c341 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075f2a85cca14f3c892ecaeba6244ea4 (127.3.75.3:39451), ceb466341f8246e990249e49378d40ba (127.3.75.1:43553)
I20260926 01:02:08.297073 18117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:08.293419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:33965 (local address 127.3.75.3:39451)
0926 01:02:08.293485 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:08.293487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:08.293499 (+ 12us) server_negotiation.cc:407] Connection header received
0926 01:02:08.293532 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:08.293533 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:08.293546 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:08.293631 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:02:08.293837 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:08.294626 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:08.296109 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:08.296684 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:08.296766 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:08.296780 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:08.296957 (+ 177us) server_negotiation.cc:299] Negotiation successful
0926 01:02:08.297001 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 01:02:08.297096 17845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:08.293570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:51331 (local address 127.3.75.1:43553)
0926 01:02:08.293635 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:08.293638 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:08.293728 (+ 90us) server_negotiation.cc:407] Connection header received
0926 01:02:08.293755 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:08.293756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:08.293797 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:08.293867 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:02:08.293997 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:08.294724 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:08.296109 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:08.296764 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:08.296830 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:08.296900 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:08.296977 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 01:02:08.297017 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 01:02:08.297075 17978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:08.293354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.3:39451 (local address 127.3.75.2:33965)
0926 01:02:08.293412 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:08.293425 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 01:02:08.293470 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:02:08.293663 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:02:08.293664 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:02:08.293666 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:02:08.293743 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:08.293745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:08.294703 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:08.294704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:08.296005 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:08.296008 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:08.296768 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:08.296769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:08.296870 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:08.296874 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:02:08.296933 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 01:02:08.296984 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":36}
I20260926 01:02:08.297080 18125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:08.293487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.1:43553 (local address 127.3.75.2:51331)
0926 01:02:08.293646 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:08.293656 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:02:08.293707 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:02:08.293880 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:02:08.293881 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:02:08.293882 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:02:08.293932 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:08.293933 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:08.294798 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:08.294798 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:08.296005 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:08.296008 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:08.296838 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:08.296839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:08.296896 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:08.296899 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:02:08.296971 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 01:02:08.296998 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":30,"thread_start_us":114,"threads_started":1}
I20260926 01:02:08.297514 17791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60612f58ad1c4ed3bc8f2f460282f4ad" candidate_uuid: "3c4cb3d7c2c749558a80ee3ad051c341" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb466341f8246e990249e49378d40ba" is_pre_election: true
I20260926 01:02:08.297518 18063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60612f58ad1c4ed3bc8f2f460282f4ad" candidate_uuid: "3c4cb3d7c2c749558a80ee3ad051c341" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075f2a85cca14f3c892ecaeba6244ea4" is_pre_election: true
I20260926 01:02:08.297680 18063 raft_consensus.cc:2454] T 60612f58ad1c4ed3bc8f2f460282f4ad P 075f2a85cca14f3c892ecaeba6244ea4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4cb3d7c2c749558a80ee3ad051c341 in term 0.
I20260926 01:02:08.297680 17791 raft_consensus.cc:2454] T 60612f58ad1c4ed3bc8f2f460282f4ad P ceb466341f8246e990249e49378d40ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4cb3d7c2c749558a80ee3ad051c341 in term 0.
I20260926 01:02:08.297928 17862 leader_election.cc:302] T 60612f58ad1c4ed3bc8f2f460282f4ad P 3c4cb3d7c2c749558a80ee3ad051c341 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4cb3d7c2c749558a80ee3ad051c341, ceb466341f8246e990249e49378d40ba; no voters:
I20260926 01:02:08.298060 18124 raft_consensus.cc:2781] T 60612f58ad1c4ed3bc8f2f460282f4ad P 3c4cb3d7c2c749558a80ee3ad051c341 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:02:08.298118 18124 raft_consensus.cc:482] T 60612f58ad1c4ed3bc8f2f460282f4ad P 3c4cb3d7c2c749558a80ee3ad051c341 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:02:08.298142 18124 raft_consensus.cc:3037] T 60612f58ad1c4ed3bc8f2f460282f4ad P 3c4cb3d7c2c749558a80ee3ad051c341 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:02:08.298828 18124 raft_consensus.cc:504] T 60612f58ad1c4ed3bc8f2f460282f4ad P 3c4cb3d7c2c749558a80ee3ad051c341 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4cb3d7c2c749558a80ee3ad051c341" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 34311 } } peers { permanent_uuid: "075f2a85cca14f3c892ecaeba6244ea4" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 39451 } } peers { permanent_uuid: "ceb466341f8246e990249e49378d40ba" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 43553 } }
I20260926 01:02:08.299029 18124 leader_election.cc:288] T 60612f58ad1c4ed3bc8f2f460282f4ad P 3c4cb3d7c2c749558a80ee3ad051c341 [CANDIDATE]: Term 1 election: Requested vote from peers 075f2a85cca14f3c892ecaeba6244ea4 (127.3.75.3:39451), ceb466341f8246e990249e49378d40ba (127.3.75.1:43553)
I20260926 01:02:08.299170 18063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60612f58ad1c4ed3bc8f2f460282f4ad" candidate_uuid: "3c4cb3d7c2c749558a80ee3ad051c341" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075f2a85cca14f3c892ecaeba6244ea4"
I20260926 01:02:08.299211 17791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60612f58ad1c4ed3bc8f2f460282f4ad" candidate_uuid: "3c4cb3d7c2c749558a80ee3ad051c341" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb466341f8246e990249e49378d40ba"
I20260926 01:02:08.299273 17791 raft_consensus.cc:3037] T 60612f58ad1c4ed3bc8f2f460282f4ad P ceb466341f8246e990249e49378d40ba [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:02:08.299274 18063 raft_consensus.cc:3037] T 60612f58ad1c4ed3bc8f2f460282f4ad P 075f2a85cca14f3c892ecaeba6244ea4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:02:08.299994 17791 raft_consensus.cc:2454] T 60612f58ad1c4ed3bc8f2f460282f4ad P ceb466341f8246e990249e49378d40ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4cb3d7c2c749558a80ee3ad051c341 in term 1.
I20260926 01:02:08.300010 18063 raft_consensus.cc:2454] T 60612f58ad1c4ed3bc8f2f460282f4ad P 075f2a85cca14f3c892ecaeba6244ea4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4cb3d7c2c749558a80ee3ad051c341 in term 1.
I20260926 01:02:08.300156 17862 leader_election.cc:302] T 60612f58ad1c4ed3bc8f2f460282f4ad P 3c4cb3d7c2c749558a80ee3ad051c341 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075f2a85cca14f3c892ecaeba6244ea4, 3c4cb3d7c2c749558a80ee3ad051c341; no voters:
I20260926 01:02:08.300293 18124 raft_consensus.cc:2781] T 60612f58ad1c4ed3bc8f2f460282f4ad P 3c4cb3d7c2c749558a80ee3ad051c341 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:02:08.300449 18124 raft_consensus.cc:686] T 60612f58ad1c4ed3bc8f2f460282f4ad P 3c4cb3d7c2c749558a80ee3ad051c341 [term 1 LEADER]: Becoming Leader. State: Replica: 3c4cb3d7c2c749558a80ee3ad051c341, State: Running, Role: LEADER
I20260926 01:02:08.300542 18124 consensus_queue.cc:237] T 60612f58ad1c4ed3bc8f2f460282f4ad P 3c4cb3d7c2c749558a80ee3ad051c341 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4cb3d7c2c749558a80ee3ad051c341" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 34311 } } peers { permanent_uuid: "075f2a85cca14f3c892ecaeba6244ea4" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 39451 } } peers { permanent_uuid: "ceb466341f8246e990249e49378d40ba" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 43553 } }
I20260926 01:02:08.301381 17648 catalog_manager.cc:5495] T 60612f58ad1c4ed3bc8f2f460282f4ad P 3c4cb3d7c2c749558a80ee3ad051c341 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c4cb3d7c2c749558a80ee3ad051c341 (127.3.75.2). New cstate: current_term: 1 leader_uuid: "3c4cb3d7c2c749558a80ee3ad051c341" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4cb3d7c2c749558a80ee3ad051c341" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 34311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "075f2a85cca14f3c892ecaeba6244ea4" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 39451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ceb466341f8246e990249e49378d40ba" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 43553 } health_report { overall_health: UNKNOWN } } }
I20260926 01:02:08.308983 18116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:08.306843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35896 (local address 127.3.75.2:34311)
0926 01:02:08.306930 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:08.306932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:08.306966 (+ 34us) server_negotiation.cc:407] Connection header received
0926 01:02:08.306994 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:08.306994 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:08.307010 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:08.307060 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:02:08.307170 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:08.307616 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:08.308296 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:08.308478 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:08.308532 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:08.308658 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:02:08.308754 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:02:08.308814 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:08.308846 (+ 32us) server_negotiation.cc:299] Negotiation successful
0926 01:02:08.308863 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 01:02:08.310731 17791 raft_consensus.cc:1261] T 60612f58ad1c4ed3bc8f2f460282f4ad P ceb466341f8246e990249e49378d40ba [term 1 FOLLOWER]: Refusing update from remote peer 3c4cb3d7c2c749558a80ee3ad051c341: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:02:08.310731 18063 raft_consensus.cc:1261] T 60612f58ad1c4ed3bc8f2f460282f4ad P 075f2a85cca14f3c892ecaeba6244ea4 [term 1 FOLLOWER]: Refusing update from remote peer 3c4cb3d7c2c749558a80ee3ad051c341: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:02:08.311113 18126 consensus_queue.cc:1035] T 60612f58ad1c4ed3bc8f2f460282f4ad P 3c4cb3d7c2c749558a80ee3ad051c341 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075f2a85cca14f3c892ecaeba6244ea4" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 39451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:02:08.311252 18124 consensus_queue.cc:1035] T 60612f58ad1c4ed3bc8f2f460282f4ad P 3c4cb3d7c2c749558a80ee3ad051c341 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb466341f8246e990249e49378d40ba" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 43553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:02:08.313373 18129 mvcc.cc:205] Tried to move back new op lower bound from 7333415027958046720 to 7333415027919663104. Current Snapshot: MvccSnapshot[applied={T|T < 7333415027958046720}]
I20260926 01:02:08.315819 17646 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:33078:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:02:08.315910 17646 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:33078:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:02:08.316565 17646 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778: Sending DeleteTablet for 3 replicas of tablet 60612f58ad1c4ed3bc8f2f460282f4ad
I20260926 01:02:08.316903 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 17686
I20260926 01:02:08.316905 17759 tablet_service.cc:1514] Processing DeleteTablet for tablet 60612f58ad1c4ed3bc8f2f460282f4ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:02:08 UTC) from {username='kudu', principal='kudu/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:58576
I20260926 01:02:08.316905 17910 tablet_service.cc:1514] Processing DeleteTablet for tablet 60612f58ad1c4ed3bc8f2f460282f4ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:02:08 UTC) from {username='kudu', principal='kudu/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:35882
I20260926 01:02:08.316984 17846 tablet_replica.cc:330] T 60612f58ad1c4ed3bc8f2f460282f4ad P ceb466341f8246e990249e49378d40ba: stopping tablet replica
I20260926 01:02:08.317029 17846 raft_consensus.cc:2229] T 60612f58ad1c4ed3bc8f2f460282f4ad P ceb466341f8246e990249e49378d40ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:02:08.317077 17846 raft_consensus.cc:2258] T 60612f58ad1c4ed3bc8f2f460282f4ad P ceb466341f8246e990249e49378d40ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:02:08.318334 18139 tablet_replica.cc:330] T 60612f58ad1c4ed3bc8f2f460282f4ad P 3c4cb3d7c2c749558a80ee3ad051c341: stopping tablet replica
I20260926 01:02:08.318431 18139 raft_consensus.cc:2229] T 60612f58ad1c4ed3bc8f2f460282f4ad P 3c4cb3d7c2c749558a80ee3ad051c341 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:02:08.318545 18139 raft_consensus.cc:2258] T 60612f58ad1c4ed3bc8f2f460282f4ad P 3c4cb3d7c2c749558a80ee3ad051c341 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:02:08.319038 18139 ts_tablet_manager.cc:1905] T 60612f58ad1c4ed3bc8f2f460282f4ad P 3c4cb3d7c2c749558a80ee3ad051c341: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:02:08.321053 18139 ts_tablet_manager.cc:1918] T 60612f58ad1c4ed3bc8f2f460282f4ad P 3c4cb3d7c2c749558a80ee3ad051c341: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:02:08.321113 18139 log.cc:1196] T 60612f58ad1c4ed3bc8f2f460282f4ad P 3c4cb3d7c2c749558a80ee3ad051c341: Deleting WAL directory at /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-1/wal/wals/60612f58ad1c4ed3bc8f2f460282f4ad
I20260926 01:02:08.321384 18139 ts_tablet_manager.cc:1939] T 60612f58ad1c4ed3bc8f2f460282f4ad P 3c4cb3d7c2c749558a80ee3ad051c341: Deleting consensus metadata
I20260926 01:02:08.321913 17632 catalog_manager.cc:4841] TS 3c4cb3d7c2c749558a80ee3ad051c341 (127.3.75.2:34311): tablet 60612f58ad1c4ed3bc8f2f460282f4ad (table test-table [id=85299fabc3284272929e1fc07125bc33]) successfully deleted
I20260926 01:02:08.322516 18043 tablet_service.cc:1514] Processing DeleteTablet for tablet 60612f58ad1c4ed3bc8f2f460282f4ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:02:08 UTC) from {username='kudu', principal='kudu/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:50946
I20260926 01:02:08.322743 18140 tablet_replica.cc:330] T 60612f58ad1c4ed3bc8f2f460282f4ad P 075f2a85cca14f3c892ecaeba6244ea4: stopping tablet replica
I20260926 01:02:08.322845 18140 raft_consensus.cc:2229] T 60612f58ad1c4ed3bc8f2f460282f4ad P 075f2a85cca14f3c892ecaeba6244ea4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:02:08.322962 18140 raft_consensus.cc:2258] T 60612f58ad1c4ed3bc8f2f460282f4ad P 075f2a85cca14f3c892ecaeba6244ea4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:02:08.323335 18140 ts_tablet_manager.cc:1905] T 60612f58ad1c4ed3bc8f2f460282f4ad P 075f2a85cca14f3c892ecaeba6244ea4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 01:02:08.323762 17631 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.75.1:43553 (error 108)
I20260926 01:02:08.324203 17844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:08.324076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.1:43553 (local address 127.0.0.1:58578)
0926 01:02:08.324122 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:08.324156 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.1:43553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 01:02:08.324316 17631 catalog_manager.cc:4584] TS ceb466341f8246e990249e49378d40ba (127.3.75.1:43553): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 60612f58ad1c4ed3bc8f2f460282f4ad: Network error: Client connection negotiation failed: client connection to 127.3.75.1:43553: connect: Connection refused (error 111)
I20260926 01:02:08.325091 18140 ts_tablet_manager.cc:1918] T 60612f58ad1c4ed3bc8f2f460282f4ad P 075f2a85cca14f3c892ecaeba6244ea4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:02:08.325109 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 17839
I20260926 01:02:08.325786 18140 log.cc:1196] T 60612f58ad1c4ed3bc8f2f460282f4ad P 075f2a85cca14f3c892ecaeba6244ea4: Deleting WAL directory at /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-2/wal/wals/60612f58ad1c4ed3bc8f2f460282f4ad
I20260926 01:02:08.326087 18140 ts_tablet_manager.cc:1939] T 60612f58ad1c4ed3bc8f2f460282f4ad P 075f2a85cca14f3c892ecaeba6244ea4: Deleting consensus metadata
I20260926 01:02:08.326483 17631 catalog_manager.cc:4841] TS 075f2a85cca14f3c892ecaeba6244ea4 (127.3.75.3:39451): tablet 60612f58ad1c4ed3bc8f2f460282f4ad (table test-table [id=85299fabc3284272929e1fc07125bc33]) successfully deleted
I20260926 01:02:08.330582 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 17979
I20260926 01:02:08.335223 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 17614
I20260926 01:02:08.341032 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:34273
--webserver_interface=127.3.75.62
--webserver_port=37489
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:37523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.75.62:34273
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:02:08.420920 18142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:02:08.421150 18142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:02:08.421187 18142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:02:08.422317 18142 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:02:08.422384 18142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:02:08.422407 18142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:02:08.422428 18142 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:02:08.422457 18142 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:02:08.423923 18142 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:37523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.75.62:34273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:34273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.75.62
--webserver_port=37489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:02:08.424188 18142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:02:08.424415 18142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:02:08.429113 18148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:02:08.429136 18149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:02:08.429242 18151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:02:08.429404 18142 server_base.cc:1034] running on GCE node
I20260926 01:02:08.429600 18142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:02:08.429836 18142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:02:08.430948 18142 hybrid_clock.cc:648] HybridClock initialized: now 1790384528430913 us; error 32 us; skew 500 ppm
Sep 26 01:02:08 dist-test-slave-7v10 krb5kdc[17062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384528, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:02:08.433277 18142 init.cc:377] Logged in from keytab as kudu/127.3.75.62@KRBTEST.COM (short username kudu)
I20260926 01:02:08.433712 18142 webserver.cc:460] Webserver started at http://127.3.75.62:37489/ using document root <none> and password file <none>
I20260926 01:02:08.433929 18142 fs_manager.cc:359] Metadata directory not provided
I20260926 01:02:08.433972 18142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:02:08.435418 18142 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:02:08.436069 18158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:02:08.436236 18142 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 01:02:08.436342 18142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/master-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/master-0/wal
uuid: "aff9338d776a42d18e1ae239f17fa778"
format_stamp: "Formatted at 2026-09-26 01:02:07 on dist-test-slave-7v10"
server_key: "560ae9ec39a670f2ef4d5ee4c143b20e"
server_key_iv: "880158daa51a878aaa41ad79645afad3"
server_key_version: "encryptionkey@0"
I20260926 01:02:08.436672 18142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:02:08.447012 18142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:02:08.448721 18142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:02:08.448900 18142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:02:08.452687 18142 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.62:34273
I20260926 01:02:08.452754 18210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.62:34273 every 8 connection(s)
I20260926 01:02:08.453112 18142 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
I20260926 01:02:08.454597 18211 sys_catalog.cc:263] Verifying existing consensus state
I20260926 01:02:08.455322 18211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778: Bootstrap starting.
I20260926 01:02:08.455780 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 18142
I20260926 01:02:08.456041 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.1:43553
--local_ip_for_outbound_sockets=127.3.75.1
--tserver_master_addrs=127.3.75.62:34273
--webserver_port=36707
--webserver_interface=127.3.75.1
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab
--principal=kudu/127.3.75.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:37523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 01:02:08.463733 18211 log.cc:824] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778: Log is configured to *not* fsync() on all Append() calls
I20260926 01:02:08.465833 18211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 01:02:08.466128 18211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778: Bootstrap complete.
I20260926 01:02:08.468331 18211 raft_consensus.cc:348] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff9338d776a42d18e1ae239f17fa778" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 34273 } }
I20260926 01:02:08.468544 18211 raft_consensus.cc:729] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: aff9338d776a42d18e1ae239f17fa778, State: Initialized, Role: FOLLOWER
I20260926 01:02:08.468719 18211 consensus_queue.cc:260] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff9338d776a42d18e1ae239f17fa778" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 34273 } }
I20260926 01:02:08.468807 18211 raft_consensus.cc:388] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:02:08.468864 18211 raft_consensus.cc:482] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:02:08.468935 18211 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778 [term 2 FOLLOWER]: Advancing to term 3
I20260926 01:02:08.469898 18211 raft_consensus.cc:504] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff9338d776a42d18e1ae239f17fa778" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 34273 } }
I20260926 01:02:08.470016 18211 leader_election.cc:302] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aff9338d776a42d18e1ae239f17fa778; no voters:
I20260926 01:02:08.470218 18211 leader_election.cc:288] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 01:02:08.470252 18215 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 01:02:08.470368 18215 raft_consensus.cc:686] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778 [term 3 LEADER]: Becoming Leader. State: Replica: aff9338d776a42d18e1ae239f17fa778, State: Running, Role: LEADER
I20260926 01:02:08.470468 18215 consensus_queue.cc:237] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff9338d776a42d18e1ae239f17fa778" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 34273 } }
I20260926 01:02:08.470503 18211 sys_catalog.cc:564] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:02:08.470712 18217 sys_catalog.cc:455] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "aff9338d776a42d18e1ae239f17fa778" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff9338d776a42d18e1ae239f17fa778" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 34273 } } }
I20260926 01:02:08.470966 18217 sys_catalog.cc:458] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778 [sys.catalog]: This master's current role is: LEADER
I20260926 01:02:08.470903 18216 sys_catalog.cc:455] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aff9338d776a42d18e1ae239f17fa778. Latest consensus state: current_term: 3 leader_uuid: "aff9338d776a42d18e1ae239f17fa778" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff9338d776a42d18e1ae239f17fa778" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 34273 } } }
I20260926 01:02:08.471211 18216 sys_catalog.cc:458] T 00000000000000000000000000000000 P aff9338d776a42d18e1ae239f17fa778 [sys.catalog]: This master's current role is: LEADER
I20260926 01:02:08.471333 18224 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:02:08.472227 18224 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:02:08.472353 18224 catalog_manager.cc:1259] Loaded cluster ID: aef9d48a0bd14f528e23b4429a7e4d14
I20260926 01:02:08.472625 18224 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260926 01:02:08.472852 18224 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f99ff24f9ad google::LogMessage::Fail() at ??:0
@ 0x7f99ff25288a google::LogMessage::SendToLog() at ??:0
@ 0x7f99ff24f4de google::LogMessage::Flush() at ??:0
@ 0x7f99ff250a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f9a071700f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f9a071798b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f9a0111d0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f9a01112954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f9a01f766db start_thread at ??:0
@ 0x7f99fdd4d61f clone at ??:0
W20260926 01:02:08.551879 18213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:02:08.552095 18213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:02:08.552132 18213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:02:08.553236 18213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:02:08.553304 18213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:02:08.553349 18213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.1
I20260926 01:02:08.554752 18213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:37523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab
--principal=kudu/127.3.75.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.1:43553
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.75.1
--webserver_port=36707
--tserver_master_addrs=127.3.75.62:34273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.1
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:02:08.555022 18213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:02:08.555255 18213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:02:08.560089 18247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:02:08.560181 18250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:02:08.560266 18213 server_base.cc:1034] running on GCE node
W20260926 01:02:08.560089 18248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:02:08.560631 18213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:02:08.560920 18213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:02:08.562067 18213 hybrid_clock.cc:648] HybridClock initialized: now 1790384528562055 us; error 28 us; skew 500 ppm
Sep 26 01:02:08 dist-test-slave-7v10 krb5kdc[17062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384528, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:02:08.564493 18213 init.cc:377] Logged in from keytab as kudu/127.3.75.1@KRBTEST.COM (short username kudu)
I20260926 01:02:08.564937 18213 webserver.cc:460] Webserver started at http://127.3.75.1:36707/ using document root <none> and password file <none>
I20260926 01:02:08.565182 18213 fs_manager.cc:359] Metadata directory not provided
I20260926 01:02:08.565249 18213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:02:08.566568 18213 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:02:08.567292 18257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:02:08.567454 18213 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 01:02:08.567553 18213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-0/wal
uuid: "ceb466341f8246e990249e49378d40ba"
format_stamp: "Formatted at 2026-09-26 01:02:07 on dist-test-slave-7v10"
server_key: "ad543bae962dbd335f350cfbe0aa0e1b"
server_key_iv: "4545a50ad287114ad5d185b87cc89683"
server_key_version: "encryptionkey@0"
I20260926 01:02:08.567873 18213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:02:08.581844 18213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:02:08.584156 18213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:02:08.584308 18213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:02:08.584544 18213 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:02:08.584970 18264 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 01:02:08.586967 18213 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 01:02:08.587023 18213 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 01:02:08.587066 18213 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 01:02:08.587644 18213 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 01:02:08.587697 18213 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 01:02:08.587764 18264 tablet_bootstrap.cc:492] T 60612f58ad1c4ed3bc8f2f460282f4ad P ceb466341f8246e990249e49378d40ba: Bootstrap starting.
I20260926 01:02:08.594796 18213 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.1:43553
I20260926 01:02:08.594962 18374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.1:43553 every 8 connection(s)
I20260926 01:02:08.595235 18213 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
I20260926 01:02:08.598853 18264 log.cc:824] T 60612f58ad1c4ed3bc8f2f460282f4ad P ceb466341f8246e990249e49378d40ba: Log is configured to *not* fsync() on all Append() calls
I20260926 01:02:08.601534 18264 tablet_bootstrap.cc:492] T 60612f58ad1c4ed3bc8f2f460282f4ad P ceb466341f8246e990249e49378d40ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 01:02:08.601848 18264 tablet_bootstrap.cc:492] T 60612f58ad1c4ed3bc8f2f460282f4ad P ceb466341f8246e990249e49378d40ba: Bootstrap complete.
I20260926 01:02:08.603040 18264 ts_tablet_manager.cc:1397] T 60612f58ad1c4ed3bc8f2f460282f4ad P ceb466341f8246e990249e49378d40ba: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.010s
I20260926 01:02:08.603320 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 18213
I20260926 01:02:08.603577 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.2:34311
--local_ip_for_outbound_sockets=127.3.75.2
--tserver_master_addrs=127.3.75.62:34273
--webserver_port=36831
--webserver_interface=127.3.75.2
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab
--principal=kudu/127.3.75.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:37523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 01:02:08.604717 18264 raft_consensus.cc:348] T 60612f58ad1c4ed3bc8f2f460282f4ad P ceb466341f8246e990249e49378d40ba [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4cb3d7c2c749558a80ee3ad051c341" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 34311 } } peers { permanent_uuid: "075f2a85cca14f3c892ecaeba6244ea4" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 39451 } } peers { permanent_uuid: "ceb466341f8246e990249e49378d40ba" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 43553 } }
I20260926 01:02:08.605046 18264 raft_consensus.cc:729] T 60612f58ad1c4ed3bc8f2f460282f4ad P ceb466341f8246e990249e49378d40ba [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceb466341f8246e990249e49378d40ba, State: Initialized, Role: FOLLOWER
I20260926 01:02:08.605189 18264 consensus_queue.cc:260] T 60612f58ad1c4ed3bc8f2f460282f4ad P ceb466341f8246e990249e49378d40ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4cb3d7c2c749558a80ee3ad051c341" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 34311 } } peers { permanent_uuid: "075f2a85cca14f3c892ecaeba6244ea4" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 39451 } } peers { permanent_uuid: "ceb466341f8246e990249e49378d40ba" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 43553 } }
I20260926 01:02:08.605463 18264 ts_tablet_manager.cc:1428] T 60612f58ad1c4ed3bc8f2f460282f4ad P ceb466341f8246e990249e49378d40ba: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20260926 01:02:08.685308 18379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:02:08.685613 18379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:02:08.685659 18379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:02:08.688002 18377 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:02:08.595888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34273 (local address 127.3.75.1:56993)
0926 01:02:08.596053 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:08.596095 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 01:02:08.596537 (+ 442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:02:08.687826 (+ 91289us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:34273: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":98,"threads_started":1}
W20260926 01:02:08.688026 18379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:02:08.688071 18379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:02:08.688131 18379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.2
W20260926 01:02:08.688295 18375 heartbeater.cc:644] Failed to heartbeat to 127.3.75.62:34273 (0 consecutive failures): Network error: Failed to ping master at 127.3.75.62:34273: Client connection negotiation failed: client connection to 127.3.75.62:34273: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 01:02:08.688666 18377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:08.688520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34273 (local address 127.3.75.1:35091)
0926 01:02:08.688577 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:08.688619 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:34273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 01:02:08.689013 18377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:08.688933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34273 (local address 127.3.75.1:51233)
0926 01:02:08.688977 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:08.688999 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:34273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 01:02:08.689160 18375 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 01:02:08.689913 18379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:37523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab
--principal=kudu/127.3.75.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.2:34311
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.75.2
--webserver_port=36831
--tserver_master_addrs=127.3.75.62:34273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.2
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:02:08.690166 18379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:02:08.690397 18379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:02:08.695339 18385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:02:08.695341 18386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:02:08.695304 18388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:02:08.695392 18379 server_base.cc:1034] running on GCE node
I20260926 01:02:08.695650 18379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:02:08.695844 18379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:02:08.697077 18379 hybrid_clock.cc:648] HybridClock initialized: now 1790384528697057 us; error 34 us; skew 500 ppm
Sep 26 01:02:08 dist-test-slave-7v10 krb5kdc[17062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384528, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:02:08.699563 18379 init.cc:377] Logged in from keytab as kudu/127.3.75.2@KRBTEST.COM (short username kudu)
I20260926 01:02:08.700012 18379 webserver.cc:460] Webserver started at http://127.3.75.2:36831/ using document root <none> and password file <none>
I20260926 01:02:08.700230 18379 fs_manager.cc:359] Metadata directory not provided
I20260926 01:02:08.700282 18379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:02:08.701558 18379 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:02:08.702118 18395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:02:08.702287 18379 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 01:02:08.702385 18379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-1/wal
uuid: "3c4cb3d7c2c749558a80ee3ad051c341"
format_stamp: "Formatted at 2026-09-26 01:02:07 on dist-test-slave-7v10"
server_key: "341c439563262cc3d449fe7d9c373127"
server_key_iv: "1855dd36ad1a53a271802b4076d2b444"
server_key_version: "encryptionkey@0"
I20260926 01:02:08.702713 18379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:02:08.709929 18379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:02:08.712188 18379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:02:08.712352 18379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:02:08.712568 18379 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:02:08.712860 18379 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:02:08.712913 18379 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:02:08.712954 18379 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:02:08.713011 18379 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:02:08.719172 18379 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.2:34311
I20260926 01:02:08.719225 18512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.2:34311 every 8 connection(s)
I20260926 01:02:08.719602 18379 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
I20260926 01:02:08.720631 18515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:08.720283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34273 (local address 127.3.75.2:50829)
0926 01:02:08.720433 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:08.720498 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:34273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":74,"threads_started":1}
W20260926 01:02:08.721046 18513 heartbeater.cc:644] Failed to heartbeat to 127.3.75.62:34273 (0 consecutive failures): Network error: Failed to ping master at 127.3.75.62:34273: Client connection negotiation failed: client connection to 127.3.75.62:34273: connect: Connection refused (error 111)
I20260926 01:02:08.721302 18515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:08.721214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34273 (local address 127.3.75.2:38619)
0926 01:02:08.721257 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:08.721284 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:34273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260926 01:02:08.721565 18515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:08.721492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34273 (local address 127.3.75.2:47151)
0926 01:02:08.721530 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:08.721551 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:34273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260926 01:02:08.721684 18513 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 01:02:08.728574 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 18379
I20260926 01:02:08.728837 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.3:39451
--local_ip_for_outbound_sockets=127.3.75.3
--tserver_master_addrs=127.3.75.62:34273
--webserver_port=32939
--webserver_interface=127.3.75.3
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab
--principal=kudu/127.3.75.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:37523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:02:08.808665 18516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:02:08.808849 18516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:02:08.808872 18516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:02:08.810009 18516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:02:08.810048 18516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:02:08.810101 18516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.3
I20260926 01:02:08.811528 18516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:37523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab
--principal=kudu/127.3.75.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.3:39451
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.75.3
--webserver_port=32939
--tserver_master_addrs=127.3.75.62:34273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.3
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:02:08.811758 18516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:02:08.811982 18516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:02:08.816993 18521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:02:08.817065 18524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:02:08.817201 18516 server_base.cc:1034] running on GCE node
W20260926 01:02:08.817009 18522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:02:08.817421 18516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:02:08.817605 18516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:02:08.818740 18516 hybrid_clock.cc:648] HybridClock initialized: now 1790384528818722 us; error 39 us; skew 500 ppm
Sep 26 01:02:08 dist-test-slave-7v10 krb5kdc[17062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384528, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:02:08.821247 18516 init.cc:377] Logged in from keytab as kudu/127.3.75.3@KRBTEST.COM (short username kudu)
I20260926 01:02:08.821653 18516 webserver.cc:460] Webserver started at http://127.3.75.3:32939/ using document root <none> and password file <none>
I20260926 01:02:08.821890 18516 fs_manager.cc:359] Metadata directory not provided
I20260926 01:02:08.821959 18516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:02:08.823276 18516 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:02:08.823958 18531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:02:08.824138 18516 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:02:08.824239 18516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-2/wal
uuid: "075f2a85cca14f3c892ecaeba6244ea4"
format_stamp: "Formatted at 2026-09-26 01:02:07 on dist-test-slave-7v10"
server_key: "82af5ba29d9171d4f288cc330359ee73"
server_key_iv: "3dd837ed4adcb4be2e765588e539c9f3"
server_key_version: "encryptionkey@0"
I20260926 01:02:08.824576 18516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:02:08.834458 18516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:02:08.836810 18516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:02:08.836975 18516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:02:08.837209 18516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:02:08.837517 18516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:02:08.837567 18516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:02:08.837606 18516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:02:08.837633 18516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:02:08.843408 18516 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.3:39451
I20260926 01:02:08.843480 18645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.3:39451 every 8 connection(s)
I20260926 01:02:08.843808 18516 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
I20260926 01:02:08.844866 18648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:08.844505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34273 (local address 127.3.75.3:59601)
0926 01:02:08.844655 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:08.844722 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:34273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":77,"threads_started":1}
W20260926 01:02:08.845255 18646 heartbeater.cc:644] Failed to heartbeat to 127.3.75.62:34273 (0 consecutive failures): Network error: Failed to ping master at 127.3.75.62:34273: Client connection negotiation failed: client connection to 127.3.75.62:34273: connect: Connection refused (error 111)
I20260926 01:02:08.845538 18648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:08.845452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34273 (local address 127.3.75.3:42311)
0926 01:02:08.845496 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:08.845521 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:34273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 01:02:08.845785 18648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:08.845731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34273 (local address 127.3.75.3:43187)
0926 01:02:08.845757 (+ 26us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:08.845774 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:34273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10}
W20260926 01:02:08.845904 18646 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 01:02:08.853790 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 18516
I20260926 01:02:09.689905 18650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:09.689641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34273 (local address 127.3.75.1:34345)
0926 01:02:09.689817 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:09.689847 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:34273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":87,"threads_started":1}
I20260926 01:02:09.722159 18651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:09.721980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34273 (local address 127.3.75.2:60531)
0926 01:02:09.722088 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:09.722119 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:34273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":54,"threads_started":1}
I20260926 01:02:09.825839 18652 raft_consensus.cc:482] T 60612f58ad1c4ed3bc8f2f460282f4ad P ceb466341f8246e990249e49378d40ba [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:02:09.825961 18652 raft_consensus.cc:504] T 60612f58ad1c4ed3bc8f2f460282f4ad P ceb466341f8246e990249e49378d40ba [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4cb3d7c2c749558a80ee3ad051c341" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 34311 } } peers { permanent_uuid: "075f2a85cca14f3c892ecaeba6244ea4" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 39451 } } peers { permanent_uuid: "ceb466341f8246e990249e49378d40ba" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 43553 } }
I20260926 01:02:09.826238 18652 leader_election.cc:288] T 60612f58ad1c4ed3bc8f2f460282f4ad P ceb466341f8246e990249e49378d40ba [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4cb3d7c2c749558a80ee3ad051c341 (127.3.75.2:34311), 075f2a85cca14f3c892ecaeba6244ea4 (127.3.75.3:39451)
Sep 26 01:02:09 dist-test-slave-7v10 krb5kdc[17062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384528, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.1@KRBTEST.COM for kudu/127.3.75.3@KRBTEST.COM
Sep 26 01:02:09 dist-test-slave-7v10 krb5kdc[17062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384528, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.1@KRBTEST.COM for kudu/127.3.75.2@KRBTEST.COM
I20260926 01:02:09.838982 18654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:09.826571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:41039 (local address 127.3.75.3:39451)
0926 01:02:09.826719 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:09.826750 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:09.826779 (+ 29us) server_negotiation.cc:407] Connection header received
0926 01:02:09.827358 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:09.827362 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:09.827428 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:09.827585 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:02:09.828670 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:09.830136 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:09.831707 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:09.832363 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:09.832481 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:09.835257 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:02:09.835262 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:02:09.835263 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:02:09.835276 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:02:09.837296 (+ 2020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:09.837698 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:09.837701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:09.837702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:09.837810 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:09.838064 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:09.838065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:09.838066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:09.838323 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:02:09.838413 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:09.838717 (+ 304us) server_negotiation.cc:299] Negotiation successful
0926 01:02:09.838856 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":63,"threads_started":1}
I20260926 01:02:09.838982 18650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:09.826434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.3:39451 (local address 127.3.75.1:41039)
0926 01:02:09.826513 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:09.826524 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:02:09.826583 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:02:09.827656 (+ 1073us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:02:09.827660 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:02:09.828259 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:02:09.828534 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:09.828542 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:09.830254 (+ 1712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:09.830255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:09.831612 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:09.831615 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:09.832466 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:09.832467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:09.832616 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:09.833247 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:02:09.833258 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:02:09.835137 (+ 1879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:02:09.837383 (+ 2246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:02:09.837388 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:02:09.837390 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:02:09.837602 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:02:09.837888 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:02:09.837888 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:02:09.837889 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:02:09.837990 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:02:09.838415 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:02:09.838422 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:02:09.838544 (+ 122us) client_negotiation.cc:769] Sending connection context
0926 01:02:09.838647 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 01:02:09.838849 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":88}
I20260926 01:02:09.839012 18655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:09.826620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:40871 (local address 127.3.75.2:34311)
0926 01:02:09.826749 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:09.826756 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:09.826783 (+ 27us) server_negotiation.cc:407] Connection header received
0926 01:02:09.827366 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:09.827369 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:09.827423 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:09.827585 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:02:09.828671 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:09.830129 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:09.831708 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:09.832363 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:09.832481 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:09.835601 (+ 3120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:02:09.835605 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:02:09.835606 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:02:09.835618 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:02:09.837470 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:09.837825 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:09.837829 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:09.837829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:09.837927 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:09.838133 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:09.838134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:09.838134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:09.838377 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:02:09.838501 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:09.838715 (+ 214us) server_negotiation.cc:299] Negotiation successful
0926 01:02:09.838866 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":55,"threads_started":1}
I20260926 01:02:09.839095 18653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:09.826446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.2:34311 (local address 127.3.75.1:40871)
0926 01:02:09.826567 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:09.826576 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:02:09.826614 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:02:09.827656 (+ 1042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:02:09.827660 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:02:09.828272 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:02:09.828534 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:09.828542 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:09.830256 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:09.830256 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:09.831612 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:09.831615 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:09.832466 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:09.832467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:09.832616 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:09.833270 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:02:09.833274 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:02:09.835498 (+ 2224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:02:09.837550 (+ 2052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:02:09.837551 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:02:09.837551 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:02:09.837742 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:02:09.838002 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:02:09.838003 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:02:09.838003 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:02:09.838070 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:02:09.838484 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:02:09.838485 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:02:09.838560 (+ 75us) client_negotiation.cc:769] Sending connection context
0926 01:02:09.838647 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 01:02:09.838960 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":178,"thread_start_us":57,"threads_started":1}
W20260926 01:02:09.840174 18259 leader_election.cc:341] T 60612f58ad1c4ed3bc8f2f460282f4ad P ceb466341f8246e990249e49378d40ba [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3c4cb3d7c2c749558a80ee3ad051c341 (127.3.75.2:34311): Not found: Tablet not found: 60612f58ad1c4ed3bc8f2f460282f4ad
W20260926 01:02:09.840253 18258 leader_election.cc:341] T 60612f58ad1c4ed3bc8f2f460282f4ad P ceb466341f8246e990249e49378d40ba [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 075f2a85cca14f3c892ecaeba6244ea4 (127.3.75.3:39451): Not found: Tablet not found: 60612f58ad1c4ed3bc8f2f460282f4ad
I20260926 01:02:09.840317 18258 leader_election.cc:302] T 60612f58ad1c4ed3bc8f2f460282f4ad P ceb466341f8246e990249e49378d40ba [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ceb466341f8246e990249e49378d40ba; no voters: 075f2a85cca14f3c892ecaeba6244ea4, 3c4cb3d7c2c749558a80ee3ad051c341
I20260926 01:02:09.840416 18652 raft_consensus.cc:2726] T 60612f58ad1c4ed3bc8f2f460282f4ad P ceb466341f8246e990249e49378d40ba [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 01:02:09.846370 18656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:09.846174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34273 (local address 127.3.75.3:44897)
0926 01:02:09.846307 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:09.846342 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:34273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":73,"threads_started":1}
I20260926 01:02:10.690552 18657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:10.690343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34273 (local address 127.3.75.1:34267)
0926 01:02:10.690464 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:10.690504 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:34273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":63,"threads_started":1}
I20260926 01:02:10.722810 18658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:10.722610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34273 (local address 127.3.75.2:43267)
0926 01:02:10.722744 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:10.722774 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:34273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":80,"threads_started":1}
I20260926 01:02:10.847066 18659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:10.846832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34273 (local address 127.3.75.3:52763)
0926 01:02:10.846998 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:10.847032 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:34273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":118,"threads_started":1}
I20260926 01:02:11.645195 18660 raft_consensus.cc:482] T 60612f58ad1c4ed3bc8f2f460282f4ad P ceb466341f8246e990249e49378d40ba [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:02:11.645296 18660 raft_consensus.cc:504] T 60612f58ad1c4ed3bc8f2f460282f4ad P ceb466341f8246e990249e49378d40ba [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4cb3d7c2c749558a80ee3ad051c341" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 34311 } } peers { permanent_uuid: "075f2a85cca14f3c892ecaeba6244ea4" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 39451 } } peers { permanent_uuid: "ceb466341f8246e990249e49378d40ba" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 43553 } }
I20260926 01:02:11.645447 18660 leader_election.cc:288] T 60612f58ad1c4ed3bc8f2f460282f4ad P ceb466341f8246e990249e49378d40ba [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4cb3d7c2c749558a80ee3ad051c341 (127.3.75.2:34311), 075f2a85cca14f3c892ecaeba6244ea4 (127.3.75.3:39451)
W20260926 01:02:11.645807 18258 leader_election.cc:341] T 60612f58ad1c4ed3bc8f2f460282f4ad P ceb466341f8246e990249e49378d40ba [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 075f2a85cca14f3c892ecaeba6244ea4 (127.3.75.3:39451): Not found: Tablet not found: 60612f58ad1c4ed3bc8f2f460282f4ad
W20260926 01:02:11.645896 18259 leader_election.cc:341] T 60612f58ad1c4ed3bc8f2f460282f4ad P ceb466341f8246e990249e49378d40ba [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3c4cb3d7c2c749558a80ee3ad051c341 (127.3.75.2:34311): Not found: Tablet not found: 60612f58ad1c4ed3bc8f2f460282f4ad
I20260926 01:02:11.646013 18259 leader_election.cc:302] T 60612f58ad1c4ed3bc8f2f460282f4ad P ceb466341f8246e990249e49378d40ba [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ceb466341f8246e990249e49378d40ba; no voters: 075f2a85cca14f3c892ecaeba6244ea4, 3c4cb3d7c2c749558a80ee3ad051c341
I20260926 01:02:11.646095 18660 raft_consensus.cc:2726] T 60612f58ad1c4ed3bc8f2f460282f4ad P ceb466341f8246e990249e49378d40ba [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 01:02:11.691284 18661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:11.691061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34273 (local address 127.3.75.1:44205)
0926 01:02:11.691188 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:11.691224 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:34273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":69,"threads_started":1}
I20260926 01:02:11.723457 18662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:11.723245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34273 (local address 127.3.75.2:38299)
0926 01:02:11.723382 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:11.723419 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:34273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":95,"threads_started":1}
I20260926 01:02:11.847671 18663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:11.847466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34273 (local address 127.3.75.3:55795)
0926 01:02:11.847601 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:11.847635 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:34273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":82,"threads_started":1}
I20260926 01:02:12.691994 18664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:12.691789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34273 (local address 127.3.75.1:34879)
0926 01:02:12.691907 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:12.691948 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:34273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":61,"threads_started":1}
I20260926 01:02:12.724072 18665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:12.723888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34273 (local address 127.3.75.2:58035)
0926 01:02:12.724005 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:12.724035 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:34273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":70,"threads_started":1}
I20260926 01:02:12.848306 18666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:12.848081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34273 (local address 127.3.75.3:34671)
0926 01:02:12.848230 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:12.848268 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:34273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":59,"threads_started":1}
I20260926 01:02:13.647400 18667 raft_consensus.cc:482] T 60612f58ad1c4ed3bc8f2f460282f4ad P ceb466341f8246e990249e49378d40ba [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:02:13.647521 18667 raft_consensus.cc:504] T 60612f58ad1c4ed3bc8f2f460282f4ad P ceb466341f8246e990249e49378d40ba [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4cb3d7c2c749558a80ee3ad051c341" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 34311 } } peers { permanent_uuid: "075f2a85cca14f3c892ecaeba6244ea4" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 39451 } } peers { permanent_uuid: "ceb466341f8246e990249e49378d40ba" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 43553 } }
I20260926 01:02:13.647737 18667 leader_election.cc:288] T 60612f58ad1c4ed3bc8f2f460282f4ad P ceb466341f8246e990249e49378d40ba [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4cb3d7c2c749558a80ee3ad051c341 (127.3.75.2:34311), 075f2a85cca14f3c892ecaeba6244ea4 (127.3.75.3:39451)
W20260926 01:02:13.648128 18259 leader_election.cc:341] T 60612f58ad1c4ed3bc8f2f460282f4ad P ceb466341f8246e990249e49378d40ba [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3c4cb3d7c2c749558a80ee3ad051c341 (127.3.75.2:34311): Not found: Tablet not found: 60612f58ad1c4ed3bc8f2f460282f4ad
W20260926 01:02:13.648201 18258 leader_election.cc:341] T 60612f58ad1c4ed3bc8f2f460282f4ad P ceb466341f8246e990249e49378d40ba [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 075f2a85cca14f3c892ecaeba6244ea4 (127.3.75.3:39451): Not found: Tablet not found: 60612f58ad1c4ed3bc8f2f460282f4ad
I20260926 01:02:13.648253 18258 leader_election.cc:302] T 60612f58ad1c4ed3bc8f2f460282f4ad P ceb466341f8246e990249e49378d40ba [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ceb466341f8246e990249e49378d40ba; no voters: 075f2a85cca14f3c892ecaeba6244ea4, 3c4cb3d7c2c749558a80ee3ad051c341
I20260926 01:02:13.648326 18667 raft_consensus.cc:2726] T 60612f58ad1c4ed3bc8f2f460282f4ad P ceb466341f8246e990249e49378d40ba [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 01:02:13.692654 18668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:13.692451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34273 (local address 127.3.75.1:36877)
0926 01:02:13.692575 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:13.692608 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:34273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":71,"threads_started":1}
I20260926 01:02:13.724615 18669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:13.724427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34273 (local address 127.3.75.2:40479)
0926 01:02:13.724539 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:13.724575 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:34273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":72,"threads_started":1}
I20260926 01:02:13.849043 18670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:13.848811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34273 (local address 127.3.75.3:48451)
0926 01:02:13.848933 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:13.848972 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:34273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":58,"threads_started":1}
I20260926 01:02:14.693361 18671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:14.693149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34273 (local address 127.3.75.1:33409)
0926 01:02:14.693273 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:14.693312 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:34273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":67,"threads_started":1}
I20260926 01:02:14.725212 18672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:14.724985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34273 (local address 127.3.75.2:44401)
0926 01:02:14.725134 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:14.725165 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:34273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":84,"threads_started":1}
I20260926 01:02:14.849699 18673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:14.849488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34273 (local address 127.3.75.3:42157)
0926 01:02:14.849604 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:14.849646 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:34273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":65,"threads_started":1}
I20260926 01:02:15.694085 18674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:15.693859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34273 (local address 127.3.75.1:45913)
0926 01:02:15.693996 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:15.694033 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:34273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":63,"threads_started":1}
I20260926 01:02:15.725855 18675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:15.725672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34273 (local address 127.3.75.2:60977)
0926 01:02:15.725791 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:15.725814 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:34273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":66,"threads_started":1}
I20260926 01:02:15.850333 18676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:15.850099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34273 (local address 127.3.75.3:34295)
0926 01:02:15.850254 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:15.850290 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:34273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":61,"threads_started":1}
I20260926 01:02:16.694792 18677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:16.694591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34273 (local address 127.3.75.1:40575)
0926 01:02:16.694715 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:16.694749 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:34273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":62,"threads_started":1}
I20260926 01:02:16.726469 18678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:16.726266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34273 (local address 127.3.75.2:38937)
0926 01:02:16.726378 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:16.726412 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:34273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":57,"threads_started":1}
I20260926 01:02:16.851133 18679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:16.850862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34273 (local address 127.3.75.3:52689)
0926 01:02:16.851048 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:16.851090 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:34273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":120,"threads_started":1}
I20260926 01:02:17.695452 18680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:17.695257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34273 (local address 127.3.75.1:32769)
0926 01:02:17.695373 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:17.695407 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:34273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":62,"threads_started":1}
I20260926 01:02:17.727098 18681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:17.726847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34273 (local address 127.3.75.2:52653)
0926 01:02:17.727011 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:17.727046 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:34273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":112,"threads_started":1}
I20260926 01:02:17.851758 18682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:17.851567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34273 (local address 127.3.75.3:56341)
0926 01:02:17.851685 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:17.851719 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:34273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":61,"threads_started":1}
I20260926 01:02:18.676409 18683 raft_consensus.cc:482] T 60612f58ad1c4ed3bc8f2f460282f4ad P ceb466341f8246e990249e49378d40ba [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:02:18.676502 18683 raft_consensus.cc:504] T 60612f58ad1c4ed3bc8f2f460282f4ad P ceb466341f8246e990249e49378d40ba [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4cb3d7c2c749558a80ee3ad051c341" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 34311 } } peers { permanent_uuid: "075f2a85cca14f3c892ecaeba6244ea4" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 39451 } } peers { permanent_uuid: "ceb466341f8246e990249e49378d40ba" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 43553 } }
I20260926 01:02:18.676679 18683 leader_election.cc:288] T 60612f58ad1c4ed3bc8f2f460282f4ad P ceb466341f8246e990249e49378d40ba [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c4cb3d7c2c749558a80ee3ad051c341 (127.3.75.2:34311), 075f2a85cca14f3c892ecaeba6244ea4 (127.3.75.3:39451)
W20260926 01:02:18.677040 18259 leader_election.cc:341] T 60612f58ad1c4ed3bc8f2f460282f4ad P ceb466341f8246e990249e49378d40ba [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3c4cb3d7c2c749558a80ee3ad051c341 (127.3.75.2:34311): Not found: Tablet not found: 60612f58ad1c4ed3bc8f2f460282f4ad
W20260926 01:02:18.677169 18258 leader_election.cc:341] T 60612f58ad1c4ed3bc8f2f460282f4ad P ceb466341f8246e990249e49378d40ba [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 075f2a85cca14f3c892ecaeba6244ea4 (127.3.75.3:39451): Not found: Tablet not found: 60612f58ad1c4ed3bc8f2f460282f4ad
I20260926 01:02:18.677229 18258 leader_election.cc:302] T 60612f58ad1c4ed3bc8f2f460282f4ad P ceb466341f8246e990249e49378d40ba [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ceb466341f8246e990249e49378d40ba; no voters: 075f2a85cca14f3c892ecaeba6244ea4, 3c4cb3d7c2c749558a80ee3ad051c341
I20260926 01:02:18.677299 18683 raft_consensus.cc:2726] T 60612f58ad1c4ed3bc8f2f460282f4ad P ceb466341f8246e990249e49378d40ba [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 01:02:18.696085 18684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:18.695885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34273 (local address 127.3.75.1:40545)
0926 01:02:18.696012 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:18.696046 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:34273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":57,"threads_started":1}
I20260926 01:02:18.727828 18685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:18.727548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34273 (local address 127.3.75.2:33517)
0926 01:02:18.727710 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:18.727750 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:34273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":76,"threads_started":1}
I20260926 01:02:18.852423 18686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:18.852199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34273 (local address 127.3.75.3:34273)
0926 01:02:18.852329 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:18.852365 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:34273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":74,"threads_started":1}
I20260926 01:02:19.696761 18687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:19.696547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34273 (local address 127.3.75.1:33335)
0926 01:02:19.696676 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:19.696707 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:34273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":72,"threads_started":1}
I20260926 01:02:19.728621 18688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:19.728364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34273 (local address 127.3.75.2:33263)
0926 01:02:19.728515 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:19.728560 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:34273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":109,"threads_started":1}
I20260926 01:02:19.853065 18689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:19.852872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34273 (local address 127.3.75.3:46959)
0926 01:02:19.852993 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:19.853023 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:34273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":65,"threads_started":1}
I20260926 01:02:20.697523 18690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:20.697212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34273 (local address 127.3.75.1:60221)
0926 01:02:20.697438 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:20.697477 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:34273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":67,"threads_started":1}
I20260926 01:02:20.729328 18691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:20.729045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34273 (local address 127.3.75.2:51055)
0926 01:02:20.729212 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:20.729262 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:34273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":90,"threads_started":1}
I20260926 01:02:20.853884 18692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:20.853588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34273 (local address 127.3.75.3:41713)
0926 01:02:20.853779 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:20.853835 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:34273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":82,"threads_started":1}
I20260926 01:02:21.698410 18693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:21.698149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34273 (local address 127.3.75.1:42603)
0926 01:02:21.698300 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:21.698350 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:34273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":96,"threads_started":1}
I20260926 01:02:21.730031 18694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:21.729820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34273 (local address 127.3.75.2:56903)
0926 01:02:21.729943 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:21.729987 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:34273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":63,"threads_started":1}
I20260926 01:02:21.854568 18695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:21.854358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34273 (local address 127.3.75.3:58895)
0926 01:02:21.854491 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:21.854525 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:34273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":75,"threads_started":1}
I20260926 01:02:22.699172 18696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:22.698946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34273 (local address 127.3.75.1:50243)
0926 01:02:22.699083 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:22.699125 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:34273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":63,"threads_started":1}
I20260926 01:02:22.730691 18697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:22.730441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34273 (local address 127.3.75.2:46269)
0926 01:02:22.730597 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:22.730638 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:34273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":60,"threads_started":1}
I20260926 01:02:22.855386 18698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:22.855156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34273 (local address 127.3.75.3:48853)
0926 01:02:22.855297 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:22.855340 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:34273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":66,"threads_started":1}
I20260926 01:02:23.699896 18699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:23.699694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34273 (local address 127.3.75.1:44999)
0926 01:02:23.699817 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:23.699853 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:34273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":60,"threads_started":1}
I20260926 01:02:23.731443 18700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:23.731262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34273 (local address 127.3.75.2:35927)
0926 01:02:23.731370 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:23.731401 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:34273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":55,"threads_started":1}
I20260926 01:02:23.856563 18701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:23.856334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34273 (local address 127.3.75.3:49571)
0926 01:02:23.856478 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:23.856514 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:34273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":70,"threads_started":1}
I20260926 01:02:23.861804 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 18213
I20260926 01:02:23.868872 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 18379
I20260926 01:02:23.874261 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 18516
2026-09-26T01:02:23Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18076 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:02:23 dist-test-slave-7v10 krb5kdc[18704](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:02:23 dist-test-slave-7v10 krb5kdc[18704](info): set up 2 sockets
Sep 26 01:02:23 dist-test-slave-7v10 krb5kdc[18704](info): commencing operation
krb5kdc: starting...
W20260926 01:02:25.002616 3372 mini_kdc.cc:121] Time spent starting KDC: real 1.108s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Sep 26 01:02:25 dist-test-slave-7v10 krb5kdc[18704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:02:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:02:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.75.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab.
Entry for principal kudu/127.3.75.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab.
I20260926 01:02:25.116041 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:44369
--webserver_interface=127.3.75.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:41541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.75.62:44369
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:02:25.196906 18717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:02:25.197101 18717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:02:25.197122 18717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:02:25.198311 18717 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 01:02:25.198370 18717 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 01:02:25.198383 18717 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 01:02:25.198395 18717 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:02:25.198444 18717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:02:25.198462 18717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:02:25.198485 18717 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:02:25.198510 18717 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:02:25.200172 18717 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:41541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.75.62:44369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:44369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:02:25.200423 18717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:02:25.200671 18717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:02:25.205957 18722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:02:25.205989 18723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:02:25.206166 18717 server_base.cc:1034] running on GCE node
W20260926 01:02:25.205966 18725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:02:25.206499 18717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:02:25.206816 18717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:02:25.207975 18717 hybrid_clock.cc:648] HybridClock initialized: now 1790384545207955 us; error 38 us; skew 500 ppm
Sep 26 01:02:25 dist-test-slave-7v10 krb5kdc[18704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384545, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:02:25.210455 18717 init.cc:377] Logged in from keytab as kudu/127.3.75.62@KRBTEST.COM (short username kudu)
I20260926 01:02:25.210904 18717 webserver.cc:460] Webserver started at http://127.3.75.62:45483/ using document root <none> and password file <none>
I20260926 01:02:25.211160 18717 fs_manager.cc:359] Metadata directory not provided
I20260926 01:02:25.211229 18717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:02:25.211378 18717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:02:25.212433 18717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/master-0/data/instance:
uuid: "fb82054a1adb4eb29679b12f92a193a4"
format_stamp: "Formatted at 2026-09-26 01:02:25 on dist-test-slave-7v10"
server_key: "ea502795c473a5210aefc564fb8774c7"
server_key_iv: "a1869dc604407891798add151e72a1c4"
server_key_version: "encryptionkey@0"
I20260926 01:02:25.212780 18717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/master-0/wal/instance:
uuid: "fb82054a1adb4eb29679b12f92a193a4"
format_stamp: "Formatted at 2026-09-26 01:02:25 on dist-test-slave-7v10"
server_key: "ea502795c473a5210aefc564fb8774c7"
server_key_iv: "a1869dc604407891798add151e72a1c4"
server_key_version: "encryptionkey@0"
I20260926 01:02:25.214109 18717 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:02:25.215065 18732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:02:25.215245 18717 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:02:25.215348 18717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/master-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/master-0/wal
uuid: "fb82054a1adb4eb29679b12f92a193a4"
format_stamp: "Formatted at 2026-09-26 01:02:25 on dist-test-slave-7v10"
server_key: "ea502795c473a5210aefc564fb8774c7"
server_key_iv: "a1869dc604407891798add151e72a1c4"
server_key_version: "encryptionkey@0"
I20260926 01:02:25.215420 18717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:02:25.233471 18717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:02:25.235576 18717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:02:25.235746 18717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:02:25.239523 18717 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.62:44369
I20260926 01:02:25.239558 18785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.62:44369 every 8 connection(s)
I20260926 01:02:25.240082 18717 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
I20260926 01:02:25.240638 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 18717
I20260926 01:02:25.240746 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/master-0/wal/instance
I20260926 01:02:25.240789 18786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:02:25.240963 3372 external_mini_cluster.cc:1428] Setting key c07a0dbfee598f0b20c5ef4ed1ad5eed
I20260926 01:02:25.243788 18786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4: Bootstrap starting.
I20260926 01:02:25.244388 18786 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4: Neither blocks nor log segments found. Creating new log.
I20260926 01:02:25.244673 18786 log.cc:824] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4: Log is configured to *not* fsync() on all Append() calls
I20260926 01:02:25.245342 18786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4: No bootstrap required, opened a new log
Sep 26 01:02:25 dist-test-slave-7v10 krb5kdc[18704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 01:02:25.248021 18786 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb82054a1adb4eb29679b12f92a193a4" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 44369 } }
I20260926 01:02:25.248189 18786 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:02:25.248252 18786 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb82054a1adb4eb29679b12f92a193a4, State: Initialized, Role: FOLLOWER
I20260926 01:02:25.248426 18786 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb82054a1adb4eb29679b12f92a193a4" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 44369 } }
I20260926 01:02:25.248521 18786 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:02:25.248566 18786 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:02:25.248615 18786 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:02:25.249163 18786 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb82054a1adb4eb29679b12f92a193a4" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 44369 } }
I20260926 01:02:25.249282 18786 leader_election.cc:302] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb82054a1adb4eb29679b12f92a193a4; no voters:
I20260926 01:02:25.249495 18786 leader_election.cc:288] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:02:25.249522 18791 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:02:25.249657 18791 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4 [term 1 LEADER]: Becoming Leader. State: Replica: fb82054a1adb4eb29679b12f92a193a4, State: Running, Role: LEADER
I20260926 01:02:25.249781 18786 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:02:25.249791 18791 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb82054a1adb4eb29679b12f92a193a4" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 44369 } }
I20260926 01:02:25.250273 18791 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb82054a1adb4eb29679b12f92a193a4. Latest consensus state: current_term: 1 leader_uuid: "fb82054a1adb4eb29679b12f92a193a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb82054a1adb4eb29679b12f92a193a4" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 44369 } } }
I20260926 01:02:25.250535 18791 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4 [sys.catalog]: This master's current role is: LEADER
I20260926 01:02:25.250456 18793 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb82054a1adb4eb29679b12f92a193a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb82054a1adb4eb29679b12f92a193a4" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 44369 } } }
I20260926 01:02:25.251004 18793 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4 [sys.catalog]: This master's current role is: LEADER
I20260926 01:02:25.251017 18800 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:02:25.251838 18800 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:02:25.252064 18789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:25.241498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57142 (local address 127.3.75.62:44369)
0926 01:02:25.241697 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:25.241707 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:25.241740 (+ 33us) server_negotiation.cc:407] Connection header received
0926 01:02:25.242278 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:25.242285 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:25.242334 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:25.242478 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:02:25.243211 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:25.244164 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:25.245033 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:25.245495 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:25.245607 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:25.248198 (+ 2591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:02:25.248202 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:02:25.248202 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:02:25.248230 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:02:25.250291 (+ 2061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:25.250741 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:25.250743 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:25.250744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:25.250842 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:25.251106 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:25.251107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:25.251107 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:25.251424 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:02:25.251617 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:25.251685 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 01:02:25.251920 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":80,"threads_started":1}
I20260926 01:02:25.253393 18800 catalog_manager.cc:1347] Generated new cluster ID: 5454e40aff7e4a7a864d614f4a70c427
I20260926 01:02:25.253449 18800 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:02:25.269688 18800 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:02:25.270408 18800 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:02:25.273099 18800 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4: Generated new TSK 0
I20260926 01:02:25.273281 18800 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.75.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab.
Entry for principal kudu/127.3.75.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab.
I20260926 01:02:25.302450 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.1:0
--local_ip_for_outbound_sockets=127.3.75.1
--webserver_interface=127.3.75.1
--webserver_port=0
--tserver_master_addrs=127.3.75.62:44369
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab
--principal=kudu/127.3.75.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:41541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:02:25.383709 18812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:02:25.383910 18812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:02:25.383934 18812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:02:25.385006 18812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:02:25.385038 18812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:02:25.385063 18812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.1
I20260926 01:02:25.386413 18812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:41541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab
--principal=kudu/127.3.75.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.75.1
--webserver_port=0
--tserver_master_addrs=127.3.75.62:44369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.1
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:02:25.386619 18812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:02:25.386842 18812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:02:25.391727 18818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:02:25.391737 18817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:02:25.391736 18820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:02:25.392091 18812 server_base.cc:1034] running on GCE node
I20260926 01:02:25.392256 18812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:02:25.392488 18812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:02:25.393626 18812 hybrid_clock.cc:648] HybridClock initialized: now 1790384545393614 us; error 29 us; skew 500 ppm
Sep 26 01:02:25 dist-test-slave-7v10 krb5kdc[18704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384545, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:02:25.395972 18812 init.cc:377] Logged in from keytab as kudu/127.3.75.1@KRBTEST.COM (short username kudu)
I20260926 01:02:25.396375 18812 webserver.cc:460] Webserver started at http://127.3.75.1:35113/ using document root <none> and password file <none>
I20260926 01:02:25.396625 18812 fs_manager.cc:359] Metadata directory not provided
I20260926 01:02:25.396693 18812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:02:25.396837 18812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:02:25.397848 18812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-0/data/instance:
uuid: "e71ada4ee8bf404997fb7bec1cef568f"
format_stamp: "Formatted at 2026-09-26 01:02:25 on dist-test-slave-7v10"
server_key: "f3e93385c9aac282dc820f306d26ec2d"
server_key_iv: "f5d21e0258108ead7bf184e414b60a56"
server_key_version: "encryptionkey@0"
I20260926 01:02:25.398201 18812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-0/wal/instance:
uuid: "e71ada4ee8bf404997fb7bec1cef568f"
format_stamp: "Formatted at 2026-09-26 01:02:25 on dist-test-slave-7v10"
server_key: "f3e93385c9aac282dc820f306d26ec2d"
server_key_iv: "f5d21e0258108ead7bf184e414b60a56"
server_key_version: "encryptionkey@0"
I20260926 01:02:25.399576 18812 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:02:25.400292 18827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:02:25.400455 18812 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:02:25.400552 18812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-0/wal
uuid: "e71ada4ee8bf404997fb7bec1cef568f"
format_stamp: "Formatted at 2026-09-26 01:02:25 on dist-test-slave-7v10"
server_key: "f3e93385c9aac282dc820f306d26ec2d"
server_key_iv: "f5d21e0258108ead7bf184e414b60a56"
server_key_version: "encryptionkey@0"
I20260926 01:02:25.400631 18812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:02:25.431278 18812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:02:25.433193 18812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:02:25.433372 18812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:02:25.433601 18812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:02:25.433904 18812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:02:25.433957 18812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:02:25.433990 18812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:02:25.434017 18812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:02:25.439563 18812 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.1:46791
I20260926 01:02:25.439626 18941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.1:46791 every 8 connection(s)
I20260926 01:02:25.439913 18812 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
Sep 26 01:02:25 dist-test-slave-7v10 krb5kdc[18704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384545, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.1@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 01:02:25.447698 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 18812
I20260926 01:02:25.447830 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-0/wal/instance
I20260926 01:02:25.448015 3372 external_mini_cluster.cc:1428] Setting key d9c319afe380e8a8f6a8251a470cc607
I20260926 01:02:25.449147 18789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:25.440561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:46841 (local address 127.3.75.62:44369)
0926 01:02:25.440623 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:25.440626 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:25.440833 (+ 207us) server_negotiation.cc:407] Connection header received
0926 01:02:25.441363 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:25.441365 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:25.441376 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:25.441429 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:02:25.442392 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:25.443017 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:25.444038 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:25.444261 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:25.444349 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:25.446463 (+ 2114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:02:25.446464 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:02:25.446465 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:02:25.446471 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:02:25.447685 (+ 1214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:25.448165 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:25.448166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:25.448166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:25.448226 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:25.448484 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:25.448485 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:25.448486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:25.448600 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:02:25.448670 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:25.448955 (+ 285us) server_negotiation.cc:299] Negotiation successful
0926 01:02:25.449046 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 01:02:25.449347 18944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:25.440541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44369 (local address 127.3.75.1:46841)
0926 01:02:25.440690 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:25.440732 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 01:02:25.441257 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:02:25.441510 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:02:25.441514 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:02:25.442082 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:02:25.442278 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:25.442283 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:25.443116 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:25.443117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:25.443941 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:25.443944 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:25.444362 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:25.444362 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:25.444493 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:25.445024 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:02:25.445032 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:02:25.446375 (+ 1343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:02:25.447770 (+ 1395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:02:25.447777 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:02:25.447779 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:02:25.448051 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:02:25.448296 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:02:25.448297 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:02:25.448298 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:02:25.448413 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:02:25.448673 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:02:25.448680 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:02:25.448819 (+ 139us) client_negotiation.cc:769] Sending connection context
0926 01:02:25.448919 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 01:02:25.449076 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":70,"threads_started":1}
I20260926 01:02:25.449828 18942 heartbeater.cc:347] Connected to a master server at 127.3.75.62:44369
I20260926 01:02:25.449939 18942 heartbeater.cc:464] Registering TS with master...
I20260926 01:02:25.450160 18942 heartbeater.cc:511] Master 127.3.75.62:44369 requested a full tablet report, sending...
I20260926 01:02:25.451581 18749 ts_manager.cc:194] Registered new tserver with Master: e71ada4ee8bf404997fb7bec1cef568f (127.3.75.1:46791)
I20260926 01:02:25.452194 18749 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.75.1@KRBTEST.COM'} at 127.3.75.1:46841
WARNING: no policy specified for kudu/127.3.75.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab.
Entry for principal kudu/127.3.75.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab.
I20260926 01:02:25.474385 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.2:0
--local_ip_for_outbound_sockets=127.3.75.2
--webserver_interface=127.3.75.2
--webserver_port=0
--tserver_master_addrs=127.3.75.62:44369
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab
--principal=kudu/127.3.75.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:41541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:02:25.558744 18947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:02:25.559010 18947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:02:25.559037 18947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:02:25.560238 18947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:02:25.560279 18947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:02:25.560331 18947 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.2
I20260926 01:02:25.561818 18947 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:41541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab
--principal=kudu/127.3.75.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.75.2
--webserver_port=0
--tserver_master_addrs=127.3.75.62:44369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.2
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:02:25.562070 18947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:02:25.562311 18947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:02:25.567297 18953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:02:25.567305 18952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:02:25.567319 18955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:02:25.567375 18947 server_base.cc:1034] running on GCE node
I20260926 01:02:25.567806 18947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:02:25.568058 18947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:02:25.569190 18947 hybrid_clock.cc:648] HybridClock initialized: now 1790384545569169 us; error 33 us; skew 500 ppm
Sep 26 01:02:25 dist-test-slave-7v10 krb5kdc[18704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384545, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:02:25.572055 18947 init.cc:377] Logged in from keytab as kudu/127.3.75.2@KRBTEST.COM (short username kudu)
I20260926 01:02:25.572486 18947 webserver.cc:460] Webserver started at http://127.3.75.2:41721/ using document root <none> and password file <none>
I20260926 01:02:25.572718 18947 fs_manager.cc:359] Metadata directory not provided
I20260926 01:02:25.572763 18947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:02:25.572911 18947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:02:25.573975 18947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-1/data/instance:
uuid: "03010feb50264e3da4b45fd1ed045323"
format_stamp: "Formatted at 2026-09-26 01:02:25 on dist-test-slave-7v10"
server_key: "e939c6341a21860f8102d7413406ee86"
server_key_iv: "d53e19836fef09d0e066c7754aa7aa11"
server_key_version: "encryptionkey@0"
I20260926 01:02:25.574354 18947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-1/wal/instance:
uuid: "03010feb50264e3da4b45fd1ed045323"
format_stamp: "Formatted at 2026-09-26 01:02:25 on dist-test-slave-7v10"
server_key: "e939c6341a21860f8102d7413406ee86"
server_key_iv: "d53e19836fef09d0e066c7754aa7aa11"
server_key_version: "encryptionkey@0"
I20260926 01:02:25.575747 18947 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 01:02:25.576450 18962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:02:25.576629 18947 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 01:02:25.576716 18947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-1/wal
uuid: "03010feb50264e3da4b45fd1ed045323"
format_stamp: "Formatted at 2026-09-26 01:02:25 on dist-test-slave-7v10"
server_key: "e939c6341a21860f8102d7413406ee86"
server_key_iv: "d53e19836fef09d0e066c7754aa7aa11"
server_key_version: "encryptionkey@0"
I20260926 01:02:25.576786 18947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:02:25.588353 18947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:02:25.590174 18947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:02:25.590315 18947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:02:25.590538 18947 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:02:25.590826 18947 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:02:25.590863 18947 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:02:25.590956 18947 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:02:25.590975 18947 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:02:25.596640 18947 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.2:46525
I20260926 01:02:25.596696 19075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.2:46525 every 8 connection(s)
I20260926 01:02:25.597020 18947 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
I20260926 01:02:25.599216 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 18947
I20260926 01:02:25.599305 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-1/wal/instance
I20260926 01:02:25.599489 3372 external_mini_cluster.cc:1428] Setting key c313ec1e300bac25ab28fd6b1e2cc4ac
Sep 26 01:02:25 dist-test-slave-7v10 krb5kdc[18704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384545, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.2@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 01:02:25.607218 18789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:25.597848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:60943 (local address 127.3.75.62:44369)
0926 01:02:25.597938 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:25.597940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:25.598052 (+ 112us) server_negotiation.cc:407] Connection header received
0926 01:02:25.598537 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:25.598539 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:25.598553 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:25.598633 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:02:25.599716 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:25.600351 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:25.601361 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:25.601581 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:25.601667 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:25.604182 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:02:25.604198 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:02:25.604199 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:02:25.604207 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:02:25.605711 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:25.606205 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:25.606206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:25.606206 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:25.606258 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:25.606539 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:25.606540 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:25.606540 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:25.606642 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:02:25.606693 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:25.607079 (+ 386us) server_negotiation.cc:299] Negotiation successful
0926 01:02:25.607153 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 01:02:25.607497 19078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:25.597749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44369 (local address 127.3.75.2:60943)
0926 01:02:25.597904 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:25.597961 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 01:02:25.598426 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:02:25.598683 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:02:25.598686 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:02:25.599379 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:02:25.599623 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:25.599628 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:25.600453 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:25.600454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:25.601268 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:25.601270 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:25.601678 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:25.601679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:25.601809 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:25.602336 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:02:25.602344 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:02:25.604079 (+ 1735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:02:25.605793 (+ 1714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:02:25.605800 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:02:25.605802 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:02:25.606122 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:02:25.606330 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:02:25.606330 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:02:25.606331 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:02:25.606467 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:02:25.606711 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:02:25.606717 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:02:25.606913 (+ 196us) client_negotiation.cc:769] Sending connection context
0926 01:02:25.607043 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 01:02:25.607217 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":90,"threads_started":1}
WARNING: no policy specified for kudu/127.3.75.3@KRBTEST.COM; defaulting to no policy
I20260926 01:02:25.608081 19076 heartbeater.cc:347] Connected to a master server at 127.3.75.62:44369
I20260926 01:02:25.608186 19076 heartbeater.cc:464] Registering TS with master...
I20260926 01:02:25.608347 19076 heartbeater.cc:511] Master 127.3.75.62:44369 requested a full tablet report, sending...
I20260926 01:02:25.608675 18749 ts_manager.cc:194] Registered new tserver with Master: 03010feb50264e3da4b45fd1ed045323 (127.3.75.2:46525)
I20260926 01:02:25.609061 18749 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.75.2@KRBTEST.COM'} at 127.3.75.2:60943
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab.
Entry for principal kudu/127.3.75.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab.
I20260926 01:02:25.625201 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.3:0
--local_ip_for_outbound_sockets=127.3.75.3
--webserver_interface=127.3.75.3
--webserver_port=0
--tserver_master_addrs=127.3.75.62:44369
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab
--principal=kudu/127.3.75.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:41541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:02:25.708997 19081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:02:25.709259 19081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:02:25.709297 19081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:02:25.710527 19081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:02:25.710590 19081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:02:25.710635 19081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.3
I20260926 01:02:25.712132 19081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:41541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab
--principal=kudu/127.3.75.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.75.3
--webserver_port=0
--tserver_master_addrs=127.3.75.62:44369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.3
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:02:25.712407 19081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:02:25.712641 19081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:02:25.717902 19089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:02:25.717903 19086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:02:25.717937 19087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:02:25.717970 19081 server_base.cc:1034] running on GCE node
I20260926 01:02:25.718348 19081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:02:25.718644 19081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:02:25.719786 19081 hybrid_clock.cc:648] HybridClock initialized: now 1790384545719773 us; error 29 us; skew 500 ppm
Sep 26 01:02:25 dist-test-slave-7v10 krb5kdc[18704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384545, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:02:25.722400 19081 init.cc:377] Logged in from keytab as kudu/127.3.75.3@KRBTEST.COM (short username kudu)
I20260926 01:02:25.722844 19081 webserver.cc:460] Webserver started at http://127.3.75.3:34887/ using document root <none> and password file <none>
I20260926 01:02:25.723166 19081 fs_manager.cc:359] Metadata directory not provided
I20260926 01:02:25.723228 19081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:02:25.723376 19081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:02:25.724427 19081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-2/data/instance:
uuid: "688277c90f2f4e7d8ab35120f5b44ee5"
format_stamp: "Formatted at 2026-09-26 01:02:25 on dist-test-slave-7v10"
server_key: "b50a7732bb3cb93557a9eae4cf4e2e19"
server_key_iv: "63ecaec34ec0115decafe500eff6bd3e"
server_key_version: "encryptionkey@0"
I20260926 01:02:25.724788 19081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-2/wal/instance:
uuid: "688277c90f2f4e7d8ab35120f5b44ee5"
format_stamp: "Formatted at 2026-09-26 01:02:25 on dist-test-slave-7v10"
server_key: "b50a7732bb3cb93557a9eae4cf4e2e19"
server_key_iv: "63ecaec34ec0115decafe500eff6bd3e"
server_key_version: "encryptionkey@0"
I20260926 01:02:25.726169 19081 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 01:02:25.726824 19096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:02:25.726998 19081 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 01:02:25.727094 19081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-2/wal
uuid: "688277c90f2f4e7d8ab35120f5b44ee5"
format_stamp: "Formatted at 2026-09-26 01:02:25 on dist-test-slave-7v10"
server_key: "b50a7732bb3cb93557a9eae4cf4e2e19"
server_key_iv: "63ecaec34ec0115decafe500eff6bd3e"
server_key_version: "encryptionkey@0"
I20260926 01:02:25.727182 19081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:02:25.740154 19081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:02:25.742468 19081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:02:25.742635 19081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:02:25.742865 19081 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:02:25.743317 19081 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:02:25.743373 19081 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:02:25.743413 19081 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:02:25.743441 19081 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:02:25.749269 19081 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.3:46657
I20260926 01:02:25.749372 19212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.3:46657 every 8 connection(s)
I20260926 01:02:25.749687 19081 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
I20260926 01:02:25.750173 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 19081
I20260926 01:02:25.750281 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-2/wal/instance
I20260926 01:02:25.750464 3372 external_mini_cluster.cc:1428] Setting key 9f205d189116931f7d83c0cee5640433
Sep 26 01:02:25 dist-test-slave-7v10 krb5kdc[18704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384545, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.3@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 01:02:25.758517 18789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:25.750501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.3:37071 (local address 127.3.75.62:44369)
0926 01:02:25.750556 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:25.750558 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:25.750795 (+ 237us) server_negotiation.cc:407] Connection header received
0926 01:02:25.751598 (+ 803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:25.751599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:25.751609 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:25.751644 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:02:25.752535 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:25.753014 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:25.753944 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:25.754125 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:25.754174 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:25.756260 (+ 2086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:02:25.756262 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:02:25.756262 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:02:25.756268 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:02:25.757413 (+ 1145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:25.757776 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:25.757776 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:25.757777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:25.757823 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:25.758027 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:25.758028 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:25.758028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:25.758130 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:02:25.758179 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:25.758400 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 01:02:25.758456 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 01:02:25.758729 19215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:25.750477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44369 (local address 127.3.75.3:37071)
0926 01:02:25.750660 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:25.750710 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 01:02:25.751491 (+ 781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:02:25.751708 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:02:25.751711 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:02:25.752274 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:02:25.752461 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:25.752465 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:25.753092 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:25.753092 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:25.753868 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:25.753870 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:25.754194 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:25.754195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:25.754288 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:25.754816 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:02:25.754825 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:02:25.756180 (+ 1355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:02:25.757499 (+ 1319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:02:25.757505 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:02:25.757507 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:02:25.757713 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:02:25.757880 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:02:25.757881 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:02:25.757881 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:02:25.757974 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:02:25.758189 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:02:25.758193 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:02:25.758296 (+ 103us) client_negotiation.cc:769] Sending connection context
0926 01:02:25.758374 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 01:02:25.758484 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":112,"threads_started":1}
I20260926 01:02:25.759262 19213 heartbeater.cc:347] Connected to a master server at 127.3.75.62:44369
I20260926 01:02:25.759356 19213 heartbeater.cc:464] Registering TS with master...
I20260926 01:02:25.759485 19213 heartbeater.cc:511] Master 127.3.75.62:44369 requested a full tablet report, sending...
I20260926 01:02:25.759788 18749 ts_manager.cc:194] Registered new tserver with Master: 688277c90f2f4e7d8ab35120f5b44ee5 (127.3.75.3:46657)
I20260926 01:02:25.760152 18749 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.75.3@KRBTEST.COM'} at 127.3.75.3:37071
I20260926 01:02:25.762251 3372 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 01:02:25.768965 18789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:25.763545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57154 (local address 127.3.75.62:44369)
0926 01:02:25.763617 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:25.763619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:25.763663 (+ 44us) server_negotiation.cc:407] Connection header received
0926 01:02:25.763744 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:25.763745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:25.763753 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:25.763798 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:02:25.764321 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:25.764770 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:25.765443 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:25.765614 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:25.765664 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:25.766685 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:02:25.766686 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:02:25.766687 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:02:25.766692 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:02:25.768014 (+ 1322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:25.768309 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:25.768309 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:25.768309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:25.768355 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:25.768504 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:25.768505 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:25.768505 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:25.768644 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:02:25.768716 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:25.768808 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 01:02:25.768865 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 01:02:25.770337 18749 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57154:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:02:25.772011 18749 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:02:25.778827 19223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:25.774431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59814 (local address 127.3.75.2:46525)
0926 01:02:25.774615 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:25.774621 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:25.774632 (+ 11us) server_negotiation.cc:407] Connection header received
0926 01:02:25.774676 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:25.774678 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:25.774696 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:25.774809 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:02:25.775016 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:25.776259 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:25.777792 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:25.778466 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:25.778532 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:25.778584 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:25.778697 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 01:02:25.778741 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":98,"threads_started":1}
I20260926 01:02:25.778837 19221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:25.774207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.2:46525 (local address 127.0.0.1:59814)
0926 01:02:25.774327 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:25.774370 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 01:02:25.774433 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:02:25.774813 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:02:25.774816 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:02:25.774818 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:02:25.774922 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:25.774925 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:25.776373 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:25.776374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:25.777704 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:25.777707 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:25.778549 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:25.778550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:25.778599 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:25.778601 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 01:02:25.778671 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 01:02:25.778738 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"mutex_wait_us":98,"thread_start_us":73,"threads_started":1}
I20260926 01:02:25.779096 19225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:25.774812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47114 (local address 127.3.75.3:46657)
0926 01:02:25.775007 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:25.775011 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:25.775021 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:02:25.775059 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:25.775060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:25.775075 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:25.775158 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:02:25.775364 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:25.776406 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:25.777618 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:25.778361 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:25.778441 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:25.778507 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:25.778971 (+ 464us) server_negotiation.cc:299] Negotiation successful
0926 01:02:25.779015 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":113,"threads_started":1}
I20260926 01:02:25.778836 19224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:25.774569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.3:46657 (local address 127.0.0.1:47114)
0926 01:02:25.774681 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:25.774691 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:02:25.774737 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:02:25.775239 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:02:25.775240 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:02:25.775242 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:02:25.775285 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:25.775286 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:25.776515 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:25.776516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:25.777535 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:25.777537 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:25.778461 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:25.778462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:25.778567 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:25.778584 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 01:02:25.778671 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 01:02:25.778738 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":70,"threads_started":1}
I20260926 01:02:25.779767 19010 tablet_service.cc:1467] Processing CreateTablet for tablet a9e9e8d87c7841458ed6f29c108a1a3c (DEFAULT_TABLE table=test-table [id=f5aa8696f2d246b9be26d8034ad21e3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:02:25.779994 19222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:25.774422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.1:46791 (local address 127.0.0.1:45786)
0926 01:02:25.774560 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:25.774572 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:02:25.774630 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:02:25.775355 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:02:25.775356 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:02:25.775358 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:02:25.775422 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:25.775423 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:25.776270 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:25.776271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:25.777705 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:25.777707 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:25.779729 (+ 2022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:25.779731 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:25.779810 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:25.779814 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:02:25.779885 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 01:02:25.779926 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":78,"thread_start_us":61,"threads_started":1}
I20260926 01:02:25.780032 19226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:25.774845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45786 (local address 127.3.75.1:46791)
0926 01:02:25.775122 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:25.775127 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:25.775137 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:02:25.775182 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:25.775185 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:25.775199 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:25.775275 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:02:25.775505 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:25.776164 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:25.778627 (+ 2463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:25.779649 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:25.779713 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:25.779754 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:25.779893 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 01:02:25.779924 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":120,"threads_started":1}
I20260926 01:02:25.780172 19010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9e9e8d87c7841458ed6f29c108a1a3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:02:25.782737 19227 tablet_bootstrap.cc:492] T a9e9e8d87c7841458ed6f29c108a1a3c P 03010feb50264e3da4b45fd1ed045323: Bootstrap starting.
I20260926 01:02:25.783277 18875 tablet_service.cc:1467] Processing CreateTablet for tablet a9e9e8d87c7841458ed6f29c108a1a3c (DEFAULT_TABLE table=test-table [id=f5aa8696f2d246b9be26d8034ad21e3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:02:25.783373 19227 tablet_bootstrap.cc:654] T a9e9e8d87c7841458ed6f29c108a1a3c P 03010feb50264e3da4b45fd1ed045323: Neither blocks nor log segments found. Creating new log.
I20260926 01:02:25.783569 18875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9e9e8d87c7841458ed6f29c108a1a3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:02:25.783892 19227 log.cc:824] T a9e9e8d87c7841458ed6f29c108a1a3c P 03010feb50264e3da4b45fd1ed045323: Log is configured to *not* fsync() on all Append() calls
I20260926 01:02:25.784579 19227 tablet_bootstrap.cc:492] T a9e9e8d87c7841458ed6f29c108a1a3c P 03010feb50264e3da4b45fd1ed045323: No bootstrap required, opened a new log
I20260926 01:02:25.784641 19227 ts_tablet_manager.cc:1397] T a9e9e8d87c7841458ed6f29c108a1a3c P 03010feb50264e3da4b45fd1ed045323: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:02:25.786152 19229 tablet_bootstrap.cc:492] T a9e9e8d87c7841458ed6f29c108a1a3c P e71ada4ee8bf404997fb7bec1cef568f: Bootstrap starting.
I20260926 01:02:25.786137 19227 raft_consensus.cc:348] T a9e9e8d87c7841458ed6f29c108a1a3c P 03010feb50264e3da4b45fd1ed045323 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03010feb50264e3da4b45fd1ed045323" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 46525 } } peers { permanent_uuid: "e71ada4ee8bf404997fb7bec1cef568f" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 46791 } } peers { permanent_uuid: "688277c90f2f4e7d8ab35120f5b44ee5" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46657 } }
I20260926 01:02:25.786355 19227 raft_consensus.cc:374] T a9e9e8d87c7841458ed6f29c108a1a3c P 03010feb50264e3da4b45fd1ed045323 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:02:25.786401 19227 raft_consensus.cc:729] T a9e9e8d87c7841458ed6f29c108a1a3c P 03010feb50264e3da4b45fd1ed045323 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03010feb50264e3da4b45fd1ed045323, State: Initialized, Role: FOLLOWER
I20260926 01:02:25.786506 19227 consensus_queue.cc:260] T a9e9e8d87c7841458ed6f29c108a1a3c P 03010feb50264e3da4b45fd1ed045323 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03010feb50264e3da4b45fd1ed045323" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 46525 } } peers { permanent_uuid: "e71ada4ee8bf404997fb7bec1cef568f" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 46791 } } peers { permanent_uuid: "688277c90f2f4e7d8ab35120f5b44ee5" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46657 } }
I20260926 01:02:25.786556 19146 tablet_service.cc:1467] Processing CreateTablet for tablet a9e9e8d87c7841458ed6f29c108a1a3c (DEFAULT_TABLE table=test-table [id=f5aa8696f2d246b9be26d8034ad21e3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:02:25.786779 19227 ts_tablet_manager.cc:1428] T a9e9e8d87c7841458ed6f29c108a1a3c P 03010feb50264e3da4b45fd1ed045323: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 01:02:25.786830 19146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9e9e8d87c7841458ed6f29c108a1a3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:02:25.786892 19229 tablet_bootstrap.cc:654] T a9e9e8d87c7841458ed6f29c108a1a3c P e71ada4ee8bf404997fb7bec1cef568f: Neither blocks nor log segments found. Creating new log.
I20260926 01:02:25.786938 19076 heartbeater.cc:503] Master 127.3.75.62:44369 was elected leader, sending a full tablet report...
I20260926 01:02:25.787386 19229 log.cc:824] T a9e9e8d87c7841458ed6f29c108a1a3c P e71ada4ee8bf404997fb7bec1cef568f: Log is configured to *not* fsync() on all Append() calls
I20260926 01:02:25.788254 19229 tablet_bootstrap.cc:492] T a9e9e8d87c7841458ed6f29c108a1a3c P e71ada4ee8bf404997fb7bec1cef568f: No bootstrap required, opened a new log
I20260926 01:02:25.788305 19229 ts_tablet_manager.cc:1397] T a9e9e8d87c7841458ed6f29c108a1a3c P e71ada4ee8bf404997fb7bec1cef568f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:02:25.789569 19232 tablet_bootstrap.cc:492] T a9e9e8d87c7841458ed6f29c108a1a3c P 688277c90f2f4e7d8ab35120f5b44ee5: Bootstrap starting.
I20260926 01:02:25.789770 19229 raft_consensus.cc:348] T a9e9e8d87c7841458ed6f29c108a1a3c P e71ada4ee8bf404997fb7bec1cef568f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03010feb50264e3da4b45fd1ed045323" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 46525 } } peers { permanent_uuid: "e71ada4ee8bf404997fb7bec1cef568f" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 46791 } } peers { permanent_uuid: "688277c90f2f4e7d8ab35120f5b44ee5" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46657 } }
I20260926 01:02:25.789974 19229 raft_consensus.cc:374] T a9e9e8d87c7841458ed6f29c108a1a3c P e71ada4ee8bf404997fb7bec1cef568f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:02:25.790007 19229 raft_consensus.cc:729] T a9e9e8d87c7841458ed6f29c108a1a3c P e71ada4ee8bf404997fb7bec1cef568f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e71ada4ee8bf404997fb7bec1cef568f, State: Initialized, Role: FOLLOWER
I20260926 01:02:25.790159 19229 consensus_queue.cc:260] T a9e9e8d87c7841458ed6f29c108a1a3c P e71ada4ee8bf404997fb7bec1cef568f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03010feb50264e3da4b45fd1ed045323" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 46525 } } peers { permanent_uuid: "e71ada4ee8bf404997fb7bec1cef568f" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 46791 } } peers { permanent_uuid: "688277c90f2f4e7d8ab35120f5b44ee5" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46657 } }
I20260926 01:02:25.790223 19232 tablet_bootstrap.cc:654] T a9e9e8d87c7841458ed6f29c108a1a3c P 688277c90f2f4e7d8ab35120f5b44ee5: Neither blocks nor log segments found. Creating new log.
I20260926 01:02:25.790450 19229 ts_tablet_manager.cc:1428] T a9e9e8d87c7841458ed6f29c108a1a3c P e71ada4ee8bf404997fb7bec1cef568f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 01:02:25.790508 18942 heartbeater.cc:503] Master 127.3.75.62:44369 was elected leader, sending a full tablet report...
I20260926 01:02:25.790779 19232 log.cc:824] T a9e9e8d87c7841458ed6f29c108a1a3c P 688277c90f2f4e7d8ab35120f5b44ee5: Log is configured to *not* fsync() on all Append() calls
I20260926 01:02:25.791668 19232 tablet_bootstrap.cc:492] T a9e9e8d87c7841458ed6f29c108a1a3c P 688277c90f2f4e7d8ab35120f5b44ee5: No bootstrap required, opened a new log
I20260926 01:02:25.791718 19232 ts_tablet_manager.cc:1397] T a9e9e8d87c7841458ed6f29c108a1a3c P 688277c90f2f4e7d8ab35120f5b44ee5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 01:02:25.792989 19232 raft_consensus.cc:348] T a9e9e8d87c7841458ed6f29c108a1a3c P 688277c90f2f4e7d8ab35120f5b44ee5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03010feb50264e3da4b45fd1ed045323" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 46525 } } peers { permanent_uuid: "e71ada4ee8bf404997fb7bec1cef568f" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 46791 } } peers { permanent_uuid: "688277c90f2f4e7d8ab35120f5b44ee5" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46657 } }
I20260926 01:02:25.793116 19232 raft_consensus.cc:374] T a9e9e8d87c7841458ed6f29c108a1a3c P 688277c90f2f4e7d8ab35120f5b44ee5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:02:25.793165 19232 raft_consensus.cc:729] T a9e9e8d87c7841458ed6f29c108a1a3c P 688277c90f2f4e7d8ab35120f5b44ee5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 688277c90f2f4e7d8ab35120f5b44ee5, State: Initialized, Role: FOLLOWER
I20260926 01:02:25.793275 19232 consensus_queue.cc:260] T a9e9e8d87c7841458ed6f29c108a1a3c P 688277c90f2f4e7d8ab35120f5b44ee5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03010feb50264e3da4b45fd1ed045323" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 46525 } } peers { permanent_uuid: "e71ada4ee8bf404997fb7bec1cef568f" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 46791 } } peers { permanent_uuid: "688277c90f2f4e7d8ab35120f5b44ee5" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46657 } }
I20260926 01:02:25.793550 19232 ts_tablet_manager.cc:1428] T a9e9e8d87c7841458ed6f29c108a1a3c P 688277c90f2f4e7d8ab35120f5b44ee5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 01:02:25.793583 19213 heartbeater.cc:503] Master 127.3.75.62:44369 was elected leader, sending a full tablet report...
W20260926 01:02:25.847538 19077 tablet.cc:2369] T a9e9e8d87c7841458ed6f29c108a1a3c P 03010feb50264e3da4b45fd1ed045323: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 01:02:25.940706 18943 tablet.cc:2369] T a9e9e8d87c7841458ed6f29c108a1a3c P e71ada4ee8bf404997fb7bec1cef568f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:02:25.985380 19230 raft_consensus.cc:482] T a9e9e8d87c7841458ed6f29c108a1a3c P 03010feb50264e3da4b45fd1ed045323 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:02:25.985539 19230 raft_consensus.cc:504] T a9e9e8d87c7841458ed6f29c108a1a3c P 03010feb50264e3da4b45fd1ed045323 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03010feb50264e3da4b45fd1ed045323" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 46525 } } peers { permanent_uuid: "e71ada4ee8bf404997fb7bec1cef568f" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 46791 } } peers { permanent_uuid: "688277c90f2f4e7d8ab35120f5b44ee5" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46657 } }
I20260926 01:02:25.985948 19230 leader_election.cc:288] T a9e9e8d87c7841458ed6f29c108a1a3c P 03010feb50264e3da4b45fd1ed045323 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e71ada4ee8bf404997fb7bec1cef568f (127.3.75.1:46791), 688277c90f2f4e7d8ab35120f5b44ee5 (127.3.75.3:46657)
I20260926 01:02:25.989315 19226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:25.986297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:52663 (local address 127.3.75.1:46791)
0926 01:02:25.986361 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:25.986363 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:25.986449 (+ 86us) server_negotiation.cc:407] Connection header received
0926 01:02:25.986624 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:25.986626 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:25.986640 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:25.986714 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:02:25.987072 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:25.987558 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:25.988585 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:25.988952 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:25.989037 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:25.989048 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:25.989202 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 01:02:25.989244 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 01:02:25.989317 19078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:25.986156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.1:46791 (local address 127.3.75.2:52663)
0926 01:02:25.986271 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:25.986368 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0926 01:02:25.986465 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:02:25.986802 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:02:25.986803 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:02:25.986804 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:02:25.986929 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:25.986931 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:25.987675 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:25.987676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:25.988486 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:25.988487 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:25.989036 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:25.989037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:25.989123 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:25.989126 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:02:25.989194 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 01:02:25.989244 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":84}
I20260926 01:02:25.989379 19225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:25.986413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:57495 (local address 127.3.75.3:46657)
0926 01:02:25.986549 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:25.986552 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:25.986605 (+ 53us) server_negotiation.cc:407] Connection header received
0926 01:02:25.986684 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:25.986686 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:25.986698 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:25.986763 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:02:25.986976 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:25.987533 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:25.988640 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:25.989015 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:25.989100 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:25.989109 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:25.989270 (+ 161us) server_negotiation.cc:299] Negotiation successful
0926 01:02:25.989312 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260926 01:02:25.989642 19236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:25.986309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.3:46657 (local address 127.3.75.2:57495)
0926 01:02:25.986537 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:25.986547 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:02:25.986605 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:02:25.986776 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:02:25.986778 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:02:25.986781 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:02:25.986876 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:25.986878 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:25.987615 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:25.987616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:25.988484 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:25.988528 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:25.989101 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:25.989101 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:25.989180 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:25.989183 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:02:25.989261 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 01:02:25.989298 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":13,"thread_start_us":118,"threads_started":1}
I20260926 01:02:25.989696 18895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9e9e8d87c7841458ed6f29c108a1a3c" candidate_uuid: "03010feb50264e3da4b45fd1ed045323" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e71ada4ee8bf404997fb7bec1cef568f" is_pre_election: true
I20260926 01:02:25.989888 18895 raft_consensus.cc:2454] T a9e9e8d87c7841458ed6f29c108a1a3c P e71ada4ee8bf404997fb7bec1cef568f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03010feb50264e3da4b45fd1ed045323 in term 0.
I20260926 01:02:25.990090 19166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9e9e8d87c7841458ed6f29c108a1a3c" candidate_uuid: "03010feb50264e3da4b45fd1ed045323" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "688277c90f2f4e7d8ab35120f5b44ee5" is_pre_election: true
I20260926 01:02:25.990146 18965 leader_election.cc:302] T a9e9e8d87c7841458ed6f29c108a1a3c P 03010feb50264e3da4b45fd1ed045323 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03010feb50264e3da4b45fd1ed045323, e71ada4ee8bf404997fb7bec1cef568f; no voters:
I20260926 01:02:25.990247 19166 raft_consensus.cc:2454] T a9e9e8d87c7841458ed6f29c108a1a3c P 688277c90f2f4e7d8ab35120f5b44ee5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03010feb50264e3da4b45fd1ed045323 in term 0.
I20260926 01:02:25.990273 19230 raft_consensus.cc:2781] T a9e9e8d87c7841458ed6f29c108a1a3c P 03010feb50264e3da4b45fd1ed045323 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:02:25.990324 19230 raft_consensus.cc:482] T a9e9e8d87c7841458ed6f29c108a1a3c P 03010feb50264e3da4b45fd1ed045323 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:02:25.990362 19230 raft_consensus.cc:3037] T a9e9e8d87c7841458ed6f29c108a1a3c P 03010feb50264e3da4b45fd1ed045323 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:02:25.991168 19230 raft_consensus.cc:504] T a9e9e8d87c7841458ed6f29c108a1a3c P 03010feb50264e3da4b45fd1ed045323 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03010feb50264e3da4b45fd1ed045323" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 46525 } } peers { permanent_uuid: "e71ada4ee8bf404997fb7bec1cef568f" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 46791 } } peers { permanent_uuid: "688277c90f2f4e7d8ab35120f5b44ee5" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46657 } }
I20260926 01:02:25.991299 19230 leader_election.cc:288] T a9e9e8d87c7841458ed6f29c108a1a3c P 03010feb50264e3da4b45fd1ed045323 [CANDIDATE]: Term 1 election: Requested vote from peers e71ada4ee8bf404997fb7bec1cef568f (127.3.75.1:46791), 688277c90f2f4e7d8ab35120f5b44ee5 (127.3.75.3:46657)
I20260926 01:02:25.991499 19166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9e9e8d87c7841458ed6f29c108a1a3c" candidate_uuid: "03010feb50264e3da4b45fd1ed045323" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "688277c90f2f4e7d8ab35120f5b44ee5"
I20260926 01:02:25.991573 18895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9e9e8d87c7841458ed6f29c108a1a3c" candidate_uuid: "03010feb50264e3da4b45fd1ed045323" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e71ada4ee8bf404997fb7bec1cef568f"
I20260926 01:02:25.991604 19166 raft_consensus.cc:3037] T a9e9e8d87c7841458ed6f29c108a1a3c P 688277c90f2f4e7d8ab35120f5b44ee5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:02:25.991660 18895 raft_consensus.cc:3037] T a9e9e8d87c7841458ed6f29c108a1a3c P e71ada4ee8bf404997fb7bec1cef568f [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:02:25.992312 19166 raft_consensus.cc:2454] T a9e9e8d87c7841458ed6f29c108a1a3c P 688277c90f2f4e7d8ab35120f5b44ee5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03010feb50264e3da4b45fd1ed045323 in term 1.
I20260926 01:02:25.992427 18895 raft_consensus.cc:2454] T a9e9e8d87c7841458ed6f29c108a1a3c P e71ada4ee8bf404997fb7bec1cef568f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03010feb50264e3da4b45fd1ed045323 in term 1.
I20260926 01:02:25.992486 18965 leader_election.cc:302] T a9e9e8d87c7841458ed6f29c108a1a3c P 03010feb50264e3da4b45fd1ed045323 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03010feb50264e3da4b45fd1ed045323, 688277c90f2f4e7d8ab35120f5b44ee5; no voters:
I20260926 01:02:25.992610 19230 raft_consensus.cc:2781] T a9e9e8d87c7841458ed6f29c108a1a3c P 03010feb50264e3da4b45fd1ed045323 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:02:25.992761 19230 raft_consensus.cc:686] T a9e9e8d87c7841458ed6f29c108a1a3c P 03010feb50264e3da4b45fd1ed045323 [term 1 LEADER]: Becoming Leader. State: Replica: 03010feb50264e3da4b45fd1ed045323, State: Running, Role: LEADER
I20260926 01:02:25.992854 19230 consensus_queue.cc:237] T a9e9e8d87c7841458ed6f29c108a1a3c P 03010feb50264e3da4b45fd1ed045323 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03010feb50264e3da4b45fd1ed045323" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 46525 } } peers { permanent_uuid: "e71ada4ee8bf404997fb7bec1cef568f" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 46791 } } peers { permanent_uuid: "688277c90f2f4e7d8ab35120f5b44ee5" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46657 } }
I20260926 01:02:25.993712 18749 catalog_manager.cc:5495] T a9e9e8d87c7841458ed6f29c108a1a3c P 03010feb50264e3da4b45fd1ed045323 reported cstate change: term changed from 0 to 1, leader changed from <none> to 03010feb50264e3da4b45fd1ed045323 (127.3.75.2). New cstate: current_term: 1 leader_uuid: "03010feb50264e3da4b45fd1ed045323" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03010feb50264e3da4b45fd1ed045323" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 46525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e71ada4ee8bf404997fb7bec1cef568f" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 46791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "688277c90f2f4e7d8ab35120f5b44ee5" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46657 } health_report { overall_health: UNKNOWN } } }
W20260926 01:02:26.001035 19214 tablet.cc:2369] T a9e9e8d87c7841458ed6f29c108a1a3c P 688277c90f2f4e7d8ab35120f5b44ee5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:02:26.004009 19223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:26.001538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59818 (local address 127.3.75.2:46525)
0926 01:02:26.001609 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:26.001611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:26.001626 (+ 15us) server_negotiation.cc:407] Connection header received
0926 01:02:26.001685 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:26.001686 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:26.001702 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:26.001760 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:02:26.001875 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:26.002378 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:26.003278 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:26.003465 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:26.003519 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:26.003628 (+ 109us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:02:26.003762 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:02:26.003823 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:26.003891 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 01:02:26.003914 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 01:02:26.005748 19166 raft_consensus.cc:1261] T a9e9e8d87c7841458ed6f29c108a1a3c P 688277c90f2f4e7d8ab35120f5b44ee5 [term 1 FOLLOWER]: Refusing update from remote peer 03010feb50264e3da4b45fd1ed045323: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:02:26.005771 18895 raft_consensus.cc:1261] T a9e9e8d87c7841458ed6f29c108a1a3c P e71ada4ee8bf404997fb7bec1cef568f [term 1 FOLLOWER]: Refusing update from remote peer 03010feb50264e3da4b45fd1ed045323: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:02:26.006112 19230 consensus_queue.cc:1035] T a9e9e8d87c7841458ed6f29c108a1a3c P 03010feb50264e3da4b45fd1ed045323 [LEADER]: Connected to new peer: Peer: permanent_uuid: "688277c90f2f4e7d8ab35120f5b44ee5" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:02:26.006265 19237 consensus_queue.cc:1035] T a9e9e8d87c7841458ed6f29c108a1a3c P 03010feb50264e3da4b45fd1ed045323 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e71ada4ee8bf404997fb7bec1cef568f" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 46791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:02:26.008948 19240 mvcc.cc:205] Tried to move back new op lower bound from 7333415100436815872 to 7333415100387311616. Current Snapshot: MvccSnapshot[applied={T|T < 7333415100436815872}]
I20260926 01:02:26.009552 19241 mvcc.cc:205] Tried to move back new op lower bound from 7333415100436815872 to 7333415100387311616. Current Snapshot: MvccSnapshot[applied={T|T < 7333415100436815872 or (T in {7333415100436815872})}]
I20260926 01:02:26.011176 18748 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:02:26.011277 18748 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:02:26.012038 18748 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4: Sending DeleteTablet for 3 replicas of tablet a9e9e8d87c7841458ed6f29c108a1a3c
I20260926 01:02:26.012344 19010 tablet_service.cc:1514] Processing DeleteTablet for tablet a9e9e8d87c7841458ed6f29c108a1a3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:02:26 UTC) from {username='kudu', principal='kudu/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:59814
I20260926 01:02:26.012398 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 18812
I20260926 01:02:26.012432 18875 tablet_service.cc:1514] Processing DeleteTablet for tablet a9e9e8d87c7841458ed6f29c108a1a3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:02:26 UTC) from {username='kudu', principal='kudu/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:45786
I20260926 01:02:26.012552 19146 tablet_service.cc:1514] Processing DeleteTablet for tablet a9e9e8d87c7841458ed6f29c108a1a3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:02:26 UTC) from {username='kudu', principal='kudu/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:47114
I20260926 01:02:26.014747 19250 tablet_replica.cc:330] T a9e9e8d87c7841458ed6f29c108a1a3c P 03010feb50264e3da4b45fd1ed045323: stopping tablet replica
I20260926 01:02:26.014856 19250 raft_consensus.cc:2229] T a9e9e8d87c7841458ed6f29c108a1a3c P 03010feb50264e3da4b45fd1ed045323 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:02:26.014981 19250 raft_consensus.cc:2258] T a9e9e8d87c7841458ed6f29c108a1a3c P 03010feb50264e3da4b45fd1ed045323 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:02:26.015432 19250 ts_tablet_manager.cc:1905] T a9e9e8d87c7841458ed6f29c108a1a3c P 03010feb50264e3da4b45fd1ed045323: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:02:26.017460 19250 ts_tablet_manager.cc:1918] T a9e9e8d87c7841458ed6f29c108a1a3c P 03010feb50264e3da4b45fd1ed045323: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:02:26.017532 19250 log.cc:1196] T a9e9e8d87c7841458ed6f29c108a1a3c P 03010feb50264e3da4b45fd1ed045323: Deleting WAL directory at /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-1/wal/wals/a9e9e8d87c7841458ed6f29c108a1a3c
I20260926 01:02:26.017792 19250 ts_tablet_manager.cc:1939] T a9e9e8d87c7841458ed6f29c108a1a3c P 03010feb50264e3da4b45fd1ed045323: Deleting consensus metadata
I20260926 01:02:26.018460 18735 catalog_manager.cc:4841] TS 03010feb50264e3da4b45fd1ed045323 (127.3.75.2:46525): tablet a9e9e8d87c7841458ed6f29c108a1a3c (table test-table [id=f5aa8696f2d246b9be26d8034ad21e3f]) successfully deleted
W20260926 01:02:26.018621 18735 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.75.1:46791 (error 108)
I20260926 01:02:26.018901 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 18947
I20260926 01:02:26.019565 19251 tablet_replica.cc:330] T a9e9e8d87c7841458ed6f29c108a1a3c P 688277c90f2f4e7d8ab35120f5b44ee5: stopping tablet replica
I20260926 01:02:26.019593 19222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:26.019011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.1:46791 (local address 127.0.0.1:45800)
0926 01:02:26.019512 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:26.019551 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.1:46791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":480}
I20260926 01:02:26.019683 19251 raft_consensus.cc:2229] T a9e9e8d87c7841458ed6f29c108a1a3c P 688277c90f2f4e7d8ab35120f5b44ee5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:02:26.019789 19251 raft_consensus.cc:2258] T a9e9e8d87c7841458ed6f29c108a1a3c P 688277c90f2f4e7d8ab35120f5b44ee5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:02:26.020233 19251 ts_tablet_manager.cc:1905] T a9e9e8d87c7841458ed6f29c108a1a3c P 688277c90f2f4e7d8ab35120f5b44ee5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:02:26.021957 19251 ts_tablet_manager.cc:1918] T a9e9e8d87c7841458ed6f29c108a1a3c P 688277c90f2f4e7d8ab35120f5b44ee5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:02:26.022022 19251 log.cc:1196] T a9e9e8d87c7841458ed6f29c108a1a3c P 688277c90f2f4e7d8ab35120f5b44ee5: Deleting WAL directory at /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-2/wal/wals/a9e9e8d87c7841458ed6f29c108a1a3c
I20260926 01:02:26.022256 19251 ts_tablet_manager.cc:1939] T a9e9e8d87c7841458ed6f29c108a1a3c P 688277c90f2f4e7d8ab35120f5b44ee5: Deleting consensus metadata
W20260926 01:02:26.023033 18735 catalog_manager.cc:4584] TS e71ada4ee8bf404997fb7bec1cef568f (127.3.75.1:46791): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a9e9e8d87c7841458ed6f29c108a1a3c: Network error: Client connection negotiation failed: client connection to 127.3.75.1:46791: connect: Connection refused (error 111)
I20260926 01:02:26.023152 18735 catalog_manager.cc:4841] TS 688277c90f2f4e7d8ab35120f5b44ee5 (127.3.75.3:46657): tablet a9e9e8d87c7841458ed6f29c108a1a3c (table test-table [id=f5aa8696f2d246b9be26d8034ad21e3f]) successfully deleted
I20260926 01:02:26.026367 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 19081
I20260926 01:02:26.031629 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 18717
I20260926 01:02:46.036996 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:44369
--webserver_interface=127.3.75.62
--webserver_port=45483
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:41541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.75.62:44369
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:02:46.115726 19253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:02:46.115950 19253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:02:46.115985 19253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:02:46.117115 19253 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 01:02:46.117180 19253 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 01:02:46.117203 19253 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 01:02:46.117223 19253 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:02:46.117254 19253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:02:46.117280 19253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:02:46.117311 19253 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:02:46.117341 19253 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:02:46.118796 19253 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:41541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.75.62:44369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:44369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.75.62
--webserver_port=45483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:02:46.119093 19253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:02:46.119330 19253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:02:46.128091 19259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:02:46.128110 19262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:02:46.128149 19253 server_base.cc:1034] running on GCE node
W20260926 01:02:46.128127 19260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:02:46.128686 19253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:02:46.128998 19253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:02:46.130172 19253 hybrid_clock.cc:648] HybridClock initialized: now 1790384566130168 us; error 35 us; skew 500 ppm
Sep 26 01:02:46 dist-test-slave-7v10 krb5kdc[18704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384566, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:02:46.132556 19253 init.cc:377] Logged in from keytab as kudu/127.3.75.62@KRBTEST.COM (short username kudu)
I20260926 01:02:46.133038 19253 webserver.cc:460] Webserver started at http://127.3.75.62:45483/ using document root <none> and password file <none>
I20260926 01:02:46.133284 19253 fs_manager.cc:359] Metadata directory not provided
I20260926 01:02:46.133368 19253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:02:46.134678 19253 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:02:46.135371 19269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:02:46.135538 19253 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 01:02:46.135636 19253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/master-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/master-0/wal
uuid: "fb82054a1adb4eb29679b12f92a193a4"
format_stamp: "Formatted at 2026-09-26 01:02:25 on dist-test-slave-7v10"
server_key: "ea502795c473a5210aefc564fb8774c7"
server_key_iv: "a1869dc604407891798add151e72a1c4"
server_key_version: "encryptionkey@0"
I20260926 01:02:46.136015 19253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:02:46.172917 19253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:02:46.174829 19253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:02:46.175022 19253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:02:46.179028 19253 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.62:44369
I20260926 01:02:46.179085 19322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.62:44369 every 8 connection(s)
I20260926 01:02:46.179513 19253 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
I20260926 01:02:46.180956 19323 sys_catalog.cc:263] Verifying existing consensus state
I20260926 01:02:46.181619 19323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4: Bootstrap starting.
I20260926 01:02:46.182196 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 19253
I20260926 01:02:46.182484 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.1:46791
--local_ip_for_outbound_sockets=127.3.75.1
--tserver_master_addrs=127.3.75.62:44369
--webserver_port=35113
--webserver_interface=127.3.75.1
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab
--principal=kudu/127.3.75.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:41541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 01:02:46.190629 19323 log.cc:824] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4: Log is configured to *not* fsync() on all Append() calls
I20260926 01:02:46.192456 19323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 01:02:46.192746 19323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4: Bootstrap complete.
I20260926 01:02:46.195106 19323 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb82054a1adb4eb29679b12f92a193a4" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 44369 } }
I20260926 01:02:46.195394 19323 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb82054a1adb4eb29679b12f92a193a4, State: Initialized, Role: FOLLOWER
I20260926 01:02:46.195591 19323 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb82054a1adb4eb29679b12f92a193a4" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 44369 } }
I20260926 01:02:46.195683 19323 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:02:46.195739 19323 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:02:46.195794 19323 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4 [term 1 FOLLOWER]: Advancing to term 2
I20260926 01:02:46.196832 19323 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb82054a1adb4eb29679b12f92a193a4" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 44369 } }
I20260926 01:02:46.196949 19323 leader_election.cc:302] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb82054a1adb4eb29679b12f92a193a4; no voters:
I20260926 01:02:46.197126 19323 leader_election.cc:288] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 01:02:46.197178 19327 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 01:02:46.197355 19327 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4 [term 2 LEADER]: Becoming Leader. State: Replica: fb82054a1adb4eb29679b12f92a193a4, State: Running, Role: LEADER
I20260926 01:02:46.197378 19323 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:02:46.197501 19327 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb82054a1adb4eb29679b12f92a193a4" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 44369 } }
I20260926 01:02:46.197753 19328 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "fb82054a1adb4eb29679b12f92a193a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb82054a1adb4eb29679b12f92a193a4" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 44369 } } }
I20260926 01:02:46.197845 19328 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4 [sys.catalog]: This master's current role is: LEADER
I20260926 01:02:46.197750 19329 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb82054a1adb4eb29679b12f92a193a4. Latest consensus state: current_term: 2 leader_uuid: "fb82054a1adb4eb29679b12f92a193a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb82054a1adb4eb29679b12f92a193a4" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 44369 } } }
I20260926 01:02:46.198226 19331 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:02:46.199082 19331 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:02:46.199263 19331 catalog_manager.cc:1259] Loaded cluster ID: 5454e40aff7e4a7a864d614f4a70c427
I20260926 01:02:46.199481 19331 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:02:46.200271 19331 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:02:46.200451 19331 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4: Loaded TSK: 0
I20260926 01:02:46.198226 19329 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4 [sys.catalog]: This master's current role is: LEADER
I20260926 01:02:46.207353 19331 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4: Generated new TSK 1
I20260926 01:02:46.207463 19331 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 01:02:46.275799 19325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:02:46.275996 19325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:02:46.276019 19325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:02:46.277182 19325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:02:46.277223 19325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:02:46.277276 19325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.1
I20260926 01:02:46.278818 19325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:41541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab
--principal=kudu/127.3.75.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.1:46791
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.75.1
--webserver_port=35113
--tserver_master_addrs=127.3.75.62:44369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.1
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:02:46.279080 19325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:02:46.279340 19325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:02:46.284222 19350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:02:46.284238 19351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:02:46.284346 19353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:02:46.284591 19325 server_base.cc:1034] running on GCE node
I20260926 01:02:46.284832 19325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:02:46.285080 19325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:02:46.286242 19325 hybrid_clock.cc:648] HybridClock initialized: now 1790384566286227 us; error 31 us; skew 500 ppm
Sep 26 01:02:46 dist-test-slave-7v10 krb5kdc[18704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384566, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:02:46.288919 19325 init.cc:377] Logged in from keytab as kudu/127.3.75.1@KRBTEST.COM (short username kudu)
I20260926 01:02:46.289371 19325 webserver.cc:460] Webserver started at http://127.3.75.1:35113/ using document root <none> and password file <none>
I20260926 01:02:46.289601 19325 fs_manager.cc:359] Metadata directory not provided
I20260926 01:02:46.289685 19325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:02:46.291054 19325 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:02:46.291791 19360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:02:46.291986 19325 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 01:02:46.292070 19325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-0/wal
uuid: "e71ada4ee8bf404997fb7bec1cef568f"
format_stamp: "Formatted at 2026-09-26 01:02:25 on dist-test-slave-7v10"
server_key: "f3e93385c9aac282dc820f306d26ec2d"
server_key_iv: "f5d21e0258108ead7bf184e414b60a56"
server_key_version: "encryptionkey@0"
I20260926 01:02:46.292440 19325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:02:46.311182 19325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:02:46.313263 19325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:02:46.313428 19325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:02:46.313661 19325 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:02:46.314165 19367 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 01:02:46.315646 19325 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 01:02:46.315696 19325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 01:02:46.315732 19325 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 01:02:46.316381 19325 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 01:02:46.316425 19325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 01:02:46.316468 19367 tablet_bootstrap.cc:492] T a9e9e8d87c7841458ed6f29c108a1a3c P e71ada4ee8bf404997fb7bec1cef568f: Bootstrap starting.
I20260926 01:02:46.324028 19325 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.1:46791
I20260926 01:02:46.324343 19475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.1:46791 every 8 connection(s)
I20260926 01:02:46.324522 19325 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
I20260926 01:02:46.327589 19367 log.cc:824] T a9e9e8d87c7841458ed6f29c108a1a3c P e71ada4ee8bf404997fb7bec1cef568f: Log is configured to *not* fsync() on all Append() calls
I20260926 01:02:46.328043 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 19325
I20260926 01:02:46.328352 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.2:46525
--local_ip_for_outbound_sockets=127.3.75.2
--tserver_master_addrs=127.3.75.62:44369
--webserver_port=41721
--webserver_interface=127.3.75.2
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab
--principal=kudu/127.3.75.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:41541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 01:02:46.330129 19367 tablet_bootstrap.cc:492] T a9e9e8d87c7841458ed6f29c108a1a3c P e71ada4ee8bf404997fb7bec1cef568f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 01:02:46.330468 19367 tablet_bootstrap.cc:492] T a9e9e8d87c7841458ed6f29c108a1a3c P e71ada4ee8bf404997fb7bec1cef568f: Bootstrap complete.
I20260926 01:02:46.331753 19367 ts_tablet_manager.cc:1397] T a9e9e8d87c7841458ed6f29c108a1a3c P e71ada4ee8bf404997fb7bec1cef568f: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.011s
I20260926 01:02:46.333181 19367 raft_consensus.cc:348] T a9e9e8d87c7841458ed6f29c108a1a3c P e71ada4ee8bf404997fb7bec1cef568f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03010feb50264e3da4b45fd1ed045323" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 46525 } } peers { permanent_uuid: "e71ada4ee8bf404997fb7bec1cef568f" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 46791 } } peers { permanent_uuid: "688277c90f2f4e7d8ab35120f5b44ee5" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46657 } }
I20260926 01:02:46.333536 19367 raft_consensus.cc:729] T a9e9e8d87c7841458ed6f29c108a1a3c P e71ada4ee8bf404997fb7bec1cef568f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e71ada4ee8bf404997fb7bec1cef568f, State: Initialized, Role: FOLLOWER
I20260926 01:02:46.333693 19367 consensus_queue.cc:260] T a9e9e8d87c7841458ed6f29c108a1a3c P e71ada4ee8bf404997fb7bec1cef568f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03010feb50264e3da4b45fd1ed045323" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 46525 } } peers { permanent_uuid: "e71ada4ee8bf404997fb7bec1cef568f" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 46791 } } peers { permanent_uuid: "688277c90f2f4e7d8ab35120f5b44ee5" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46657 } }
I20260926 01:02:46.333992 19367 ts_tablet_manager.cc:1428] T a9e9e8d87c7841458ed6f29c108a1a3c P e71ada4ee8bf404997fb7bec1cef568f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
Sep 26 01:02:46 dist-test-slave-7v10 krb5kdc[18704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384566, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.1@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 01:02:46.339805 19479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:46.325440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:33905 (local address 127.3.75.62:44369)
0926 01:02:46.325622 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:46.325631 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:46.327169 (+ 1538us) server_negotiation.cc:407] Connection header received
0926 01:02:46.327823 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:46.327831 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:46.329259 (+ 1428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:46.329385 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:02:46.330284 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:46.331867 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:46.333351 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:46.333996 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:46.334078 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:46.336376 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:02:46.336386 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:02:46.336386 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:02:46.336396 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:02:46.338024 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:46.338416 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:46.338420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:46.338421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:46.338517 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:46.338792 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:46.338793 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:46.338794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:46.339092 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:02:46.339191 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:46.339487 (+ 296us) server_negotiation.cc:299] Negotiation successful
0926 01:02:46.339614 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":80,"threads_started":1}
I20260926 01:02:46.339818 19478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:46.325375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44369 (local address 127.3.75.1:33905)
0926 01:02:46.326987 (+ 1612us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:46.327031 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 01:02:46.327590 (+ 559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:02:46.329572 (+ 1982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:02:46.329576 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:02:46.329919 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:02:46.330155 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:46.330160 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:46.332005 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:46.332006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:46.333251 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:46.333253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:46.334194 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:46.334195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:46.334285 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:46.334915 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:02:46.334925 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:02:46.336266 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:02:46.338102 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:02:46.338108 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:02:46.338110 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:02:46.338332 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:02:46.338591 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:02:46.338592 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:02:46.338592 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:02:46.338715 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:02:46.339170 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:02:46.339176 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:02:46.339321 (+ 145us) client_negotiation.cc:769] Sending connection context
0926 01:02:46.339433 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 01:02:46.339614 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1566,"thread_start_us":108,"threads_started":1}
I20260926 01:02:46.340821 19476 heartbeater.cc:347] Connected to a master server at 127.3.75.62:44369
I20260926 01:02:46.340914 19476 heartbeater.cc:464] Registering TS with master...
I20260926 01:02:46.341070 19476 heartbeater.cc:511] Master 127.3.75.62:44369 requested a full tablet report, sending...
I20260926 01:02:46.341706 19287 ts_manager.cc:194] Registered new tserver with Master: e71ada4ee8bf404997fb7bec1cef568f (127.3.75.1:46791)
I20260926 01:02:46.342782 19287 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.75.1@KRBTEST.COM'} at 127.3.75.1:33905
I20260926 01:02:46.343711 19476 heartbeater.cc:503] Master 127.3.75.62:44369 was elected leader, sending a full tablet report...
I20260926 01:02:46.350019 19484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:46.342503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49040 (local address 127.3.75.1:46791)
0926 01:02:46.347229 (+ 4726us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:46.347233 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:46.347240 (+ 7us) server_negotiation.cc:407] Connection header received
0926 01:02:46.347264 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:46.347265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:46.347277 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:46.347362 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:02:46.347483 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:46.348129 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:46.349131 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:46.349588 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:46.349671 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:46.349722 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:46.349887 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 01:02:46.349930 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4701,"thread_start_us":60,"threads_started":1}
I20260926 01:02:46.350010 19483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:46.342272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.1:46791 (local address 127.0.0.1:49040)
0926 01:02:46.347029 (+ 4757us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:46.347059 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 01:02:46.347115 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:02:46.347366 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:02:46.347368 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:02:46.347370 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:02:46.347427 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:46.347428 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:46.348233 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:46.348234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:46.349025 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:46.349027 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:46.349686 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:46.349687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:46.349769 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:46.349781 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 01:02:46.349862 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 01:02:46.349921 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4730,"thread_start_us":91,"threads_started":1}
I20260926 01:02:46.350600 19391 tablet_service.cc:1514] Processing DeleteTablet for tablet a9e9e8d87c7841458ed6f29c108a1a3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:02:26 UTC) from {username='kudu', principal='kudu/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:49040
I20260926 01:02:46.351215 19485 tablet_replica.cc:330] stopping tablet replica
I20260926 01:02:46.351331 19485 raft_consensus.cc:2229] T a9e9e8d87c7841458ed6f29c108a1a3c P e71ada4ee8bf404997fb7bec1cef568f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:02:46.351406 19485 raft_consensus.cc:2258] T a9e9e8d87c7841458ed6f29c108a1a3c P e71ada4ee8bf404997fb7bec1cef568f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:02:46.351694 19485 ts_tablet_manager.cc:1905] T a9e9e8d87c7841458ed6f29c108a1a3c P e71ada4ee8bf404997fb7bec1cef568f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:02:46.354100 19485 ts_tablet_manager.cc:1918] T a9e9e8d87c7841458ed6f29c108a1a3c P e71ada4ee8bf404997fb7bec1cef568f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:02:46.354182 19485 log.cc:1196] T a9e9e8d87c7841458ed6f29c108a1a3c P e71ada4ee8bf404997fb7bec1cef568f: Deleting WAL directory at /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-0/wal/wals/a9e9e8d87c7841458ed6f29c108a1a3c
I20260926 01:02:46.354416 19485 ts_tablet_manager.cc:1939] T a9e9e8d87c7841458ed6f29c108a1a3c P e71ada4ee8bf404997fb7bec1cef568f: Deleting consensus metadata
I20260926 01:02:46.355042 19273 catalog_manager.cc:4841] TS e71ada4ee8bf404997fb7bec1cef568f (127.3.75.1:46791): tablet a9e9e8d87c7841458ed6f29c108a1a3c (table test-table [id=f5aa8696f2d246b9be26d8034ad21e3f]) successfully deleted
W20260926 01:02:46.416827 19481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:02:46.417021 19481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:02:46.417043 19481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:02:46.418217 19481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:02:46.418258 19481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:02:46.418309 19481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.2
I20260926 01:02:46.419829 19481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:41541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab
--principal=kudu/127.3.75.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.2:46525
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.75.2
--webserver_port=41721
--tserver_master_addrs=127.3.75.62:44369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.2
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:02:46.420068 19481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:02:46.420292 19481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:02:46.425194 19490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:02:46.425199 19491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:02:46.425226 19493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:02:46.425405 19481 server_base.cc:1034] running on GCE node
I20260926 01:02:46.425630 19481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:02:46.425894 19481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:02:46.427054 19481 hybrid_clock.cc:648] HybridClock initialized: now 1790384566427032 us; error 39 us; skew 500 ppm
Sep 26 01:02:46 dist-test-slave-7v10 krb5kdc[18704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384566, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:02:46.429369 19481 init.cc:377] Logged in from keytab as kudu/127.3.75.2@KRBTEST.COM (short username kudu)
I20260926 01:02:46.429791 19481 webserver.cc:460] Webserver started at http://127.3.75.2:41721/ using document root <none> and password file <none>
I20260926 01:02:46.430016 19481 fs_manager.cc:359] Metadata directory not provided
I20260926 01:02:46.430090 19481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:02:46.431504 19481 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:02:46.432054 19500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:02:46.432248 19481 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:02:46.432331 19481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-1/wal
uuid: "03010feb50264e3da4b45fd1ed045323"
format_stamp: "Formatted at 2026-09-26 01:02:25 on dist-test-slave-7v10"
server_key: "e939c6341a21860f8102d7413406ee86"
server_key_iv: "d53e19836fef09d0e066c7754aa7aa11"
server_key_version: "encryptionkey@0"
I20260926 01:02:46.432626 19481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:02:46.447391 19481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:02:46.449560 19481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:02:46.449667 19481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:02:46.449824 19481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:02:46.450121 19481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:02:46.450155 19481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:02:46.450207 19481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:02:46.450237 19481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:02:46.455933 19481 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.2:46525
I20260926 01:02:46.456025 19614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.2:46525 every 8 connection(s)
I20260926 01:02:46.456364 19481 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
Sep 26 01:02:46 dist-test-slave-7v10 krb5kdc[18704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384566, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.2@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 01:02:46.465095 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 19481
I20260926 01:02:46.465389 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.3:46657
--local_ip_for_outbound_sockets=127.3.75.3
--tserver_master_addrs=127.3.75.62:44369
--webserver_port=34887
--webserver_interface=127.3.75.3
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab
--principal=kudu/127.3.75.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:41541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 01:02:46.465665 19479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:46.457136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:46783 (local address 127.3.75.62:44369)
0926 01:02:46.457216 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:46.457218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:46.457400 (+ 182us) server_negotiation.cc:407] Connection header received
0926 01:02:46.457917 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:46.457918 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:46.457932 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:46.457973 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:02:46.458991 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:46.459516 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:46.460557 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:46.460795 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:46.460875 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:46.463002 (+ 2127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:02:46.463003 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:02:46.463003 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:02:46.463010 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:02:46.464311 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:46.464666 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:46.464666 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:46.464667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:46.464716 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:46.464951 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:46.464951 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:46.464952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:46.465050 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:02:46.465094 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:46.465468 (+ 374us) server_negotiation.cc:299] Negotiation successful
0926 01:02:46.465575 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 01:02:46.465878 19617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:46.457108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44369 (local address 127.3.75.2:46783)
0926 01:02:46.457284 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:46.457317 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 01:02:46.457805 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:02:46.458053 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:02:46.458056 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:02:46.458648 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:02:46.458832 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:46.458837 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:46.459622 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:46.459623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:46.460464 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:46.460466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:46.460894 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:46.460895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:46.461013 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:46.461477 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:02:46.461486 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:02:46.462860 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:02:46.464379 (+ 1519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:02:46.464384 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:02:46.464387 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:02:46.464597 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:02:46.464773 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:02:46.464773 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:02:46.464774 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:02:46.464865 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:02:46.465110 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:02:46.465130 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:02:46.465305 (+ 175us) client_negotiation.cc:769] Sending connection context
0926 01:02:46.465428 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 01:02:46.465607 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":77,"threads_started":1}
I20260926 01:02:46.466427 19615 heartbeater.cc:347] Connected to a master server at 127.3.75.62:44369
I20260926 01:02:46.466538 19615 heartbeater.cc:464] Registering TS with master...
I20260926 01:02:46.466722 19615 heartbeater.cc:511] Master 127.3.75.62:44369 requested a full tablet report, sending...
I20260926 01:02:46.467178 19287 ts_manager.cc:194] Registered new tserver with Master: 03010feb50264e3da4b45fd1ed045323 (127.3.75.2:46525)
I20260926 01:02:46.467551 19287 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.75.2@KRBTEST.COM'} at 127.3.75.2:46783
W20260926 01:02:46.546525 19618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:02:46.546712 19618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:02:46.546736 19618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:02:46.547875 19618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:02:46.547919 19618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:02:46.547973 19618 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.3
I20260926 01:02:46.549360 19618 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:41541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab
--principal=kudu/127.3.75.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.3:46657
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.75.3
--webserver_port=34887
--tserver_master_addrs=127.3.75.62:44369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.3
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:02:46.549600 19618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:02:46.549844 19618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:02:46.554697 19624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:02:46.554708 19623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:02:46.554708 19626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:02:46.554996 19618 server_base.cc:1034] running on GCE node
I20260926 01:02:46.555195 19618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:02:46.555441 19618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:02:46.556600 19618 hybrid_clock.cc:648] HybridClock initialized: now 1790384566556523 us; error 105 us; skew 500 ppm
Sep 26 01:02:46 dist-test-slave-7v10 krb5kdc[18704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384566, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:02:46.558862 19618 init.cc:377] Logged in from keytab as kudu/127.3.75.3@KRBTEST.COM (short username kudu)
I20260926 01:02:46.559381 19618 webserver.cc:460] Webserver started at http://127.3.75.3:34887/ using document root <none> and password file <none>
I20260926 01:02:46.559620 19618 fs_manager.cc:359] Metadata directory not provided
I20260926 01:02:46.559708 19618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:02:46.561019 19618 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:02:46.561625 19633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:02:46.561782 19618 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 01:02:46.561894 19618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-2/wal
uuid: "688277c90f2f4e7d8ab35120f5b44ee5"
format_stamp: "Formatted at 2026-09-26 01:02:25 on dist-test-slave-7v10"
server_key: "b50a7732bb3cb93557a9eae4cf4e2e19"
server_key_iv: "63ecaec34ec0115decafe500eff6bd3e"
server_key_version: "encryptionkey@0"
I20260926 01:02:46.562222 19618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:02:46.582015 19618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:02:46.583818 19618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:02:46.583931 19618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:02:46.584105 19618 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:02:46.584403 19618 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:02:46.584435 19618 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:02:46.584481 19618 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:02:46.584508 19618 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:02:46.590354 19618 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.3:46657
I20260926 01:02:46.590413 19747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.3:46657 every 8 connection(s)
I20260926 01:02:46.590749 19618 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
I20260926 01:02:46.591216 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 19618
Sep 26 01:02:46 dist-test-slave-7v10 krb5kdc[18704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384566, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.3@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 01:02:46.598476 19752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:46.591665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37614 (local address 127.3.75.62:44369)
0926 01:02:46.591777 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:46.591779 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:46.591816 (+ 37us) server_negotiation.cc:407] Connection header received
0926 01:02:46.591903 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:46.591904 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:46.591916 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:46.591974 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:02:46.592576 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:46.593100 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:46.593923 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:46.594134 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:46.594218 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:46.595534 (+ 1316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:02:46.595535 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:02:46.595535 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:02:46.595543 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:02:46.597198 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:46.597524 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:46.597525 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:46.597526 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:46.597598 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:46.597782 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:46.597784 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:46.597784 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:46.597930 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:02:46.598076 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:46.598244 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 01:02:46.598339 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":61,"threads_started":1}
I20260926 01:02:46.600383 19479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:46.591436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.3:46637 (local address 127.3.75.62:44369)
0926 01:02:46.591507 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:46.591510 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:46.591762 (+ 252us) server_negotiation.cc:407] Connection header received
0926 01:02:46.592279 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:46.592280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:46.592292 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:46.592342 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:02:46.593249 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:46.593725 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:46.594838 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:46.595076 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:46.595186 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:46.597849 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:02:46.597850 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:02:46.597850 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:02:46.597856 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:02:46.599399 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:46.599746 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:46.599747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:46.599747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:46.599795 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:46.599960 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:46.599961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:46.599961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:46.600048 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:02:46.600094 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:46.600277 (+ 183us) server_negotiation.cc:299] Negotiation successful
0926 01:02:46.600329 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 01:02:46.600597 19750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:46.591446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44369 (local address 127.3.75.3:46637)
0926 01:02:46.591614 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:46.591670 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 01:02:46.592174 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:02:46.592413 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:02:46.592416 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:02:46.592968 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:02:46.593157 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:46.593161 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:46.593823 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:46.593824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:46.594757 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:46.594762 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:46.595198 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:46.595199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:46.595334 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:46.595835 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:02:46.595844 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:02:46.597769 (+ 1925us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:02:46.599455 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:02:46.599460 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:02:46.599462 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:02:46.599683 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:02:46.599834 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:02:46.599834 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:02:46.599835 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:02:46.599913 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:02:46.600086 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:02:46.600089 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:02:46.600193 (+ 104us) client_negotiation.cc:769] Sending connection context
0926 01:02:46.600267 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 01:02:46.600377 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":97,"threads_started":1}
I20260926 01:02:46.601073 19748 heartbeater.cc:347] Connected to a master server at 127.3.75.62:44369
I20260926 01:02:46.601172 19748 heartbeater.cc:464] Registering TS with master...
I20260926 01:02:46.601363 19748 heartbeater.cc:511] Master 127.3.75.62:44369 requested a full tablet report, sending...
I20260926 01:02:46.601737 19287 ts_manager.cc:194] Registered new tserver with Master: 688277c90f2f4e7d8ab35120f5b44ee5 (127.3.75.3:46657)
I20260926 01:02:46.602094 19287 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.75.3@KRBTEST.COM'} at 127.3.75.3:46637
I20260926 01:02:46.610045 3372 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 01:02:46.612914 19479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:46.610397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37618 (local address 127.3.75.62:44369)
0926 01:02:46.610467 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:46.610470 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:46.610594 (+ 124us) server_negotiation.cc:407] Connection header received
0926 01:02:46.610678 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:46.610679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:46.610720 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:46.610795 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:02:46.610965 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:46.611427 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:46.612210 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:46.612399 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:46.612471 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:46.612637 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:02:46.612690 (+ 53us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0926 01:02:46.612844 (+ 154us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37618: token expired
Metrics: {"server-negotiator.queue_time_us":26}
W20260926 01:02:46.612975 19479 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37618: token expired
W20260926 01:02:46.613163 19217 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.3.75.62:44369)
I20260926 01:02:46.618983 19479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:46.613454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37630 (local address 127.3.75.62:44369)
0926 01:02:46.613490 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:46.613492 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:46.613500 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:02:46.613520 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:46.613521 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:46.613529 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:46.613576 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:02:46.614058 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:46.614493 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:46.615259 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:46.615435 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:46.615503 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:46.616678 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:02:46.616679 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:02:46.616680 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:02:46.616687 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:02:46.617860 (+ 1173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:46.618189 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:46.618190 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:46.618190 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:46.618250 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:02:46.618449 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:02:46.618450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:02:46.618450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:02:46.618579 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:02:46.618694 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:46.618790 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 01:02:46.618866 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 01:02:46.619978 19287 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37630:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:02:46.621476 19287 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:02:46.623862 19391 tablet_service.cc:1467] Processing CreateTablet for tablet 3ee5d737ccb14630aca97e1b007dddb9 (DEFAULT_TABLE table=test-table [id=1e9276e93c564b66bef6778c7959fc37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:02:46.624073 19391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ee5d737ccb14630aca97e1b007dddb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:02:46.625735 19367 tablet_bootstrap.cc:492] T 3ee5d737ccb14630aca97e1b007dddb9 P e71ada4ee8bf404997fb7bec1cef568f: Bootstrap starting.
I20260926 01:02:46.626104 19367 tablet_bootstrap.cc:654] T 3ee5d737ccb14630aca97e1b007dddb9 P e71ada4ee8bf404997fb7bec1cef568f: Neither blocks nor log segments found. Creating new log.
I20260926 01:02:46.626775 19367 tablet_bootstrap.cc:492] T 3ee5d737ccb14630aca97e1b007dddb9 P e71ada4ee8bf404997fb7bec1cef568f: No bootstrap required, opened a new log
I20260926 01:02:46.626828 19367 ts_tablet_manager.cc:1397] T 3ee5d737ccb14630aca97e1b007dddb9 P e71ada4ee8bf404997fb7bec1cef568f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 01:02:46.627005 19367 raft_consensus.cc:348] T 3ee5d737ccb14630aca97e1b007dddb9 P e71ada4ee8bf404997fb7bec1cef568f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688277c90f2f4e7d8ab35120f5b44ee5" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46657 } } peers { permanent_uuid: "03010feb50264e3da4b45fd1ed045323" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 46525 } } peers { permanent_uuid: "e71ada4ee8bf404997fb7bec1cef568f" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 46791 } }
I20260926 01:02:46.627094 19367 raft_consensus.cc:374] T 3ee5d737ccb14630aca97e1b007dddb9 P e71ada4ee8bf404997fb7bec1cef568f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:02:46.627151 19367 raft_consensus.cc:729] T 3ee5d737ccb14630aca97e1b007dddb9 P e71ada4ee8bf404997fb7bec1cef568f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e71ada4ee8bf404997fb7bec1cef568f, State: Initialized, Role: FOLLOWER
I20260926 01:02:46.627193 19367 consensus_queue.cc:260] T 3ee5d737ccb14630aca97e1b007dddb9 P e71ada4ee8bf404997fb7bec1cef568f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688277c90f2f4e7d8ab35120f5b44ee5" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46657 } } peers { permanent_uuid: "03010feb50264e3da4b45fd1ed045323" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 46525 } } peers { permanent_uuid: "e71ada4ee8bf404997fb7bec1cef568f" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 46791 } }
I20260926 01:02:46.627292 19367 ts_tablet_manager.cc:1428] T 3ee5d737ccb14630aca97e1b007dddb9 P e71ada4ee8bf404997fb7bec1cef568f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 01:02:46.627544 19754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:46.623448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.2:46525 (local address 127.0.0.1:49524)
0926 01:02:46.623611 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:46.623620 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:02:46.623659 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:02:46.623918 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:02:46.623919 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:02:46.623921 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:02:46.623968 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:46.623969 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:46.625151 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:46.625152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:46.626156 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:46.626158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:46.627311 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:46.627312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:46.627380 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:46.627384 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:02:46.627436 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 01:02:46.627477 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":115,"threads_started":1}
I20260926 01:02:46.627573 19756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:46.623559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49524 (local address 127.3.75.2:46525)
0926 01:02:46.623757 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:46.623762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:46.623772 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:02:46.623812 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:46.623813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:46.623825 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:46.623922 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:02:46.624045 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:46.625056 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:46.626251 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:46.627130 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:46.627214 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:46.627276 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:46.627452 (+ 176us) server_negotiation.cc:299] Negotiation successful
0926 01:02:46.627489 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":71,"threads_started":1}
I20260926 01:02:46.628420 19755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:46.623469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39032 (local address 127.3.75.3:46657)
0926 01:02:46.624084 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:46.624089 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:46.624098 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:02:46.624133 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:46.624135 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:46.624149 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:46.624255 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:02:46.624402 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:46.625416 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:46.627365 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:46.628123 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:46.628198 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:46.628235 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:46.628334 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 01:02:46.628363 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":99,"threads_started":1}
I20260926 01:02:46.628438 19483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:46.623352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.3:46657 (local address 127.0.0.1:39032)
0926 01:02:46.623428 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:46.623441 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 01:02:46.623490 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:02:46.624264 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:02:46.624265 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:02:46.624267 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:02:46.624305 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:46.624306 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:46.625505 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:46.625506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:46.626536 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:46.626538 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:46.628200 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:46.628201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:46.628261 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:46.628264 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:02:46.628326 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 01:02:46.628369 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 01:02:46.628448 19548 tablet_service.cc:1467] Processing CreateTablet for tablet 3ee5d737ccb14630aca97e1b007dddb9 (DEFAULT_TABLE table=test-table [id=1e9276e93c564b66bef6778c7959fc37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:02:46.628746 19548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ee5d737ccb14630aca97e1b007dddb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:02:46.629360 19681 tablet_service.cc:1467] Processing CreateTablet for tablet 3ee5d737ccb14630aca97e1b007dddb9 (DEFAULT_TABLE table=test-table [id=1e9276e93c564b66bef6778c7959fc37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:02:46.629662 19681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ee5d737ccb14630aca97e1b007dddb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:02:46.632272 19758 tablet_bootstrap.cc:492] T 3ee5d737ccb14630aca97e1b007dddb9 P 03010feb50264e3da4b45fd1ed045323: Bootstrap starting.
I20260926 01:02:46.632839 19758 tablet_bootstrap.cc:654] T 3ee5d737ccb14630aca97e1b007dddb9 P 03010feb50264e3da4b45fd1ed045323: Neither blocks nor log segments found. Creating new log.
I20260926 01:02:46.632925 19759 tablet_bootstrap.cc:492] T 3ee5d737ccb14630aca97e1b007dddb9 P 688277c90f2f4e7d8ab35120f5b44ee5: Bootstrap starting.
I20260926 01:02:46.633397 19758 log.cc:824] T 3ee5d737ccb14630aca97e1b007dddb9 P 03010feb50264e3da4b45fd1ed045323: Log is configured to *not* fsync() on all Append() calls
I20260926 01:02:46.633514 19759 tablet_bootstrap.cc:654] T 3ee5d737ccb14630aca97e1b007dddb9 P 688277c90f2f4e7d8ab35120f5b44ee5: Neither blocks nor log segments found. Creating new log.
I20260926 01:02:46.634093 19759 log.cc:824] T 3ee5d737ccb14630aca97e1b007dddb9 P 688277c90f2f4e7d8ab35120f5b44ee5: Log is configured to *not* fsync() on all Append() calls
I20260926 01:02:46.634140 19758 tablet_bootstrap.cc:492] T 3ee5d737ccb14630aca97e1b007dddb9 P 03010feb50264e3da4b45fd1ed045323: No bootstrap required, opened a new log
I20260926 01:02:46.634205 19758 ts_tablet_manager.cc:1397] T 3ee5d737ccb14630aca97e1b007dddb9 P 03010feb50264e3da4b45fd1ed045323: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:02:46.634771 19759 tablet_bootstrap.cc:492] T 3ee5d737ccb14630aca97e1b007dddb9 P 688277c90f2f4e7d8ab35120f5b44ee5: No bootstrap required, opened a new log
I20260926 01:02:46.634850 19759 ts_tablet_manager.cc:1397] T 3ee5d737ccb14630aca97e1b007dddb9 P 688277c90f2f4e7d8ab35120f5b44ee5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:02:46.636057 19758 raft_consensus.cc:348] T 3ee5d737ccb14630aca97e1b007dddb9 P 03010feb50264e3da4b45fd1ed045323 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688277c90f2f4e7d8ab35120f5b44ee5" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46657 } } peers { permanent_uuid: "03010feb50264e3da4b45fd1ed045323" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 46525 } } peers { permanent_uuid: "e71ada4ee8bf404997fb7bec1cef568f" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 46791 } }
I20260926 01:02:46.636202 19758 raft_consensus.cc:374] T 3ee5d737ccb14630aca97e1b007dddb9 P 03010feb50264e3da4b45fd1ed045323 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:02:46.636224 19758 raft_consensus.cc:729] T 3ee5d737ccb14630aca97e1b007dddb9 P 03010feb50264e3da4b45fd1ed045323 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03010feb50264e3da4b45fd1ed045323, State: Initialized, Role: FOLLOWER
I20260926 01:02:46.636368 19758 consensus_queue.cc:260] T 3ee5d737ccb14630aca97e1b007dddb9 P 03010feb50264e3da4b45fd1ed045323 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688277c90f2f4e7d8ab35120f5b44ee5" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46657 } } peers { permanent_uuid: "03010feb50264e3da4b45fd1ed045323" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 46525 } } peers { permanent_uuid: "e71ada4ee8bf404997fb7bec1cef568f" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 46791 } }
I20260926 01:02:46.636479 19759 raft_consensus.cc:348] T 3ee5d737ccb14630aca97e1b007dddb9 P 688277c90f2f4e7d8ab35120f5b44ee5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688277c90f2f4e7d8ab35120f5b44ee5" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46657 } } peers { permanent_uuid: "03010feb50264e3da4b45fd1ed045323" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 46525 } } peers { permanent_uuid: "e71ada4ee8bf404997fb7bec1cef568f" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 46791 } }
I20260926 01:02:46.636619 19758 ts_tablet_manager.cc:1428] T 3ee5d737ccb14630aca97e1b007dddb9 P 03010feb50264e3da4b45fd1ed045323: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 01:02:46.636643 19615 heartbeater.cc:503] Master 127.3.75.62:44369 was elected leader, sending a full tablet report...
I20260926 01:02:46.636656 19759 raft_consensus.cc:374] T 3ee5d737ccb14630aca97e1b007dddb9 P 688277c90f2f4e7d8ab35120f5b44ee5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:02:46.636693 19759 raft_consensus.cc:729] T 3ee5d737ccb14630aca97e1b007dddb9 P 688277c90f2f4e7d8ab35120f5b44ee5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 688277c90f2f4e7d8ab35120f5b44ee5, State: Initialized, Role: FOLLOWER
I20260926 01:02:46.636793 19759 consensus_queue.cc:260] T 3ee5d737ccb14630aca97e1b007dddb9 P 688277c90f2f4e7d8ab35120f5b44ee5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688277c90f2f4e7d8ab35120f5b44ee5" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46657 } } peers { permanent_uuid: "03010feb50264e3da4b45fd1ed045323" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 46525 } } peers { permanent_uuid: "e71ada4ee8bf404997fb7bec1cef568f" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 46791 } }
I20260926 01:02:46.637063 19759 ts_tablet_manager.cc:1428] T 3ee5d737ccb14630aca97e1b007dddb9 P 688277c90f2f4e7d8ab35120f5b44ee5: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 01:02:46.637075 19748 heartbeater.cc:503] Master 127.3.75.62:44369 was elected leader, sending a full tablet report...
W20260926 01:02:46.706972 19616 tablet.cc:2369] T 3ee5d737ccb14630aca97e1b007dddb9 P 03010feb50264e3da4b45fd1ed045323: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:02:46.738900 19762 raft_consensus.cc:482] T 3ee5d737ccb14630aca97e1b007dddb9 P 03010feb50264e3da4b45fd1ed045323 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:02:46.739080 19762 raft_consensus.cc:504] T 3ee5d737ccb14630aca97e1b007dddb9 P 03010feb50264e3da4b45fd1ed045323 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688277c90f2f4e7d8ab35120f5b44ee5" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46657 } } peers { permanent_uuid: "03010feb50264e3da4b45fd1ed045323" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 46525 } } peers { permanent_uuid: "e71ada4ee8bf404997fb7bec1cef568f" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 46791 } }
I20260926 01:02:46.739547 19762 leader_election.cc:288] T 3ee5d737ccb14630aca97e1b007dddb9 P 03010feb50264e3da4b45fd1ed045323 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 688277c90f2f4e7d8ab35120f5b44ee5 (127.3.75.3:46657), e71ada4ee8bf404997fb7bec1cef568f (127.3.75.1:46791)
I20260926 01:02:46.742934 19764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:46.739867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.1:46791 (local address 127.3.75.2:52429)
0926 01:02:46.740003 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:46.740011 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 01:02:46.740049 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:02:46.740254 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:02:46.740256 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:02:46.740258 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:02:46.740336 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:46.740338 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:46.741165 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:46.741166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:46.741947 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:46.741949 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:46.742679 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:46.742680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:46.742729 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:46.742733 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:02:46.742799 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 01:02:46.742834 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":81,"threads_started":1}
I20260926 01:02:46.742928 19484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:46.740013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:52429 (local address 127.3.75.1:46791)
0926 01:02:46.740065 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:46.740067 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:46.740079 (+ 12us) server_negotiation.cc:407] Connection header received
0926 01:02:46.740109 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:46.740110 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:46.740132 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:46.740194 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:02:46.740425 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:46.741084 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:46.742037 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:46.742587 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:46.742662 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:46.742672 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:46.742805 (+ 133us) server_negotiation.cc:299] Negotiation successful
0926 01:02:46.742846 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 01:02:46.743256 19430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ee5d737ccb14630aca97e1b007dddb9" candidate_uuid: "03010feb50264e3da4b45fd1ed045323" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e71ada4ee8bf404997fb7bec1cef568f" is_pre_election: true
I20260926 01:02:46.743311 19617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:46.739743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.3:46657 (local address 127.3.75.2:53381)
0926 01:02:46.739791 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:46.739805 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 01:02:46.739875 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:02:46.740053 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:02:46.740054 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:02:46.740056 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:02:46.740126 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:46.740127 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:46.740985 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:46.740986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:46.742234 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:02:46.742236 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:02:46.743020 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:02:46.743021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:02:46.743107 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:46.743111 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:02:46.743173 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 01:02:46.743217 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260926 01:02:46.743417 19430 raft_consensus.cc:2454] T 3ee5d737ccb14630aca97e1b007dddb9 P e71ada4ee8bf404997fb7bec1cef568f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03010feb50264e3da4b45fd1ed045323 in term 0.
I20260926 01:02:46.743602 19755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:46.739770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:53381 (local address 127.3.75.3:46657)
0926 01:02:46.739817 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:46.739819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:46.739904 (+ 85us) server_negotiation.cc:407] Connection header received
0926 01:02:46.739934 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:46.739935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:46.739946 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:46.739999 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:02:46.740200 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:46.740894 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:46.742316 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:46.742905 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:46.742976 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:46.742986 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:46.743475 (+ 489us) server_negotiation.cc:299] Negotiation successful
0926 01:02:46.743523 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 01:02:46.743628 19503 leader_election.cc:302] T 3ee5d737ccb14630aca97e1b007dddb9 P 03010feb50264e3da4b45fd1ed045323 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03010feb50264e3da4b45fd1ed045323, e71ada4ee8bf404997fb7bec1cef568f; no voters:
I20260926 01:02:46.743847 19762 raft_consensus.cc:2781] T 3ee5d737ccb14630aca97e1b007dddb9 P 03010feb50264e3da4b45fd1ed045323 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:02:46.743950 19762 raft_consensus.cc:482] T 3ee5d737ccb14630aca97e1b007dddb9 P 03010feb50264e3da4b45fd1ed045323 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:02:46.743975 19762 raft_consensus.cc:3037] T 3ee5d737ccb14630aca97e1b007dddb9 P 03010feb50264e3da4b45fd1ed045323 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:02:46.744026 19701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ee5d737ccb14630aca97e1b007dddb9" candidate_uuid: "03010feb50264e3da4b45fd1ed045323" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "688277c90f2f4e7d8ab35120f5b44ee5" is_pre_election: true
I20260926 01:02:46.744189 19701 raft_consensus.cc:2454] T 3ee5d737ccb14630aca97e1b007dddb9 P 688277c90f2f4e7d8ab35120f5b44ee5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03010feb50264e3da4b45fd1ed045323 in term 0.
I20260926 01:02:46.744798 19762 raft_consensus.cc:504] T 3ee5d737ccb14630aca97e1b007dddb9 P 03010feb50264e3da4b45fd1ed045323 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688277c90f2f4e7d8ab35120f5b44ee5" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46657 } } peers { permanent_uuid: "03010feb50264e3da4b45fd1ed045323" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 46525 } } peers { permanent_uuid: "e71ada4ee8bf404997fb7bec1cef568f" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 46791 } }
I20260926 01:02:46.744925 19762 leader_election.cc:288] T 3ee5d737ccb14630aca97e1b007dddb9 P 03010feb50264e3da4b45fd1ed045323 [CANDIDATE]: Term 1 election: Requested vote from peers 688277c90f2f4e7d8ab35120f5b44ee5 (127.3.75.3:46657), e71ada4ee8bf404997fb7bec1cef568f (127.3.75.1:46791)
I20260926 01:02:46.745059 19701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ee5d737ccb14630aca97e1b007dddb9" candidate_uuid: "03010feb50264e3da4b45fd1ed045323" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "688277c90f2f4e7d8ab35120f5b44ee5"
I20260926 01:02:46.745157 19701 raft_consensus.cc:3037] T 3ee5d737ccb14630aca97e1b007dddb9 P 688277c90f2f4e7d8ab35120f5b44ee5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:02:46.745144 19430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ee5d737ccb14630aca97e1b007dddb9" candidate_uuid: "03010feb50264e3da4b45fd1ed045323" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e71ada4ee8bf404997fb7bec1cef568f"
I20260926 01:02:46.745230 19430 raft_consensus.cc:3037] T 3ee5d737ccb14630aca97e1b007dddb9 P e71ada4ee8bf404997fb7bec1cef568f [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:02:46.745734 19701 raft_consensus.cc:2454] T 3ee5d737ccb14630aca97e1b007dddb9 P 688277c90f2f4e7d8ab35120f5b44ee5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03010feb50264e3da4b45fd1ed045323 in term 1.
I20260926 01:02:46.745879 19503 leader_election.cc:302] T 3ee5d737ccb14630aca97e1b007dddb9 P 03010feb50264e3da4b45fd1ed045323 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03010feb50264e3da4b45fd1ed045323, 688277c90f2f4e7d8ab35120f5b44ee5; no voters:
I20260926 01:02:46.745947 19762 raft_consensus.cc:2781] T 3ee5d737ccb14630aca97e1b007dddb9 P 03010feb50264e3da4b45fd1ed045323 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:02:46.746089 19762 raft_consensus.cc:686] T 3ee5d737ccb14630aca97e1b007dddb9 P 03010feb50264e3da4b45fd1ed045323 [term 1 LEADER]: Becoming Leader. State: Replica: 03010feb50264e3da4b45fd1ed045323, State: Running, Role: LEADER
I20260926 01:02:46.746171 19762 consensus_queue.cc:237] T 3ee5d737ccb14630aca97e1b007dddb9 P 03010feb50264e3da4b45fd1ed045323 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688277c90f2f4e7d8ab35120f5b44ee5" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46657 } } peers { permanent_uuid: "03010feb50264e3da4b45fd1ed045323" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 46525 } } peers { permanent_uuid: "e71ada4ee8bf404997fb7bec1cef568f" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 46791 } }
I20260926 01:02:46.746568 19430 raft_consensus.cc:2454] T 3ee5d737ccb14630aca97e1b007dddb9 P e71ada4ee8bf404997fb7bec1cef568f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03010feb50264e3da4b45fd1ed045323 in term 1.
I20260926 01:02:46.747071 19286 catalog_manager.cc:5495] T 3ee5d737ccb14630aca97e1b007dddb9 P 03010feb50264e3da4b45fd1ed045323 reported cstate change: term changed from 0 to 1, leader changed from <none> to 03010feb50264e3da4b45fd1ed045323 (127.3.75.2). New cstate: current_term: 1 leader_uuid: "03010feb50264e3da4b45fd1ed045323" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688277c90f2f4e7d8ab35120f5b44ee5" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03010feb50264e3da4b45fd1ed045323" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 46525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e71ada4ee8bf404997fb7bec1cef568f" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 46791 } health_report { overall_health: UNKNOWN } } }
I20260926 01:02:46.771400 19756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:46.769026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49540 (local address 127.3.75.2:46525)
0926 01:02:46.769086 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 01:02:46.769088 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:02:46.769097 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:02:46.769188 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:02:46.769189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:02:46.769207 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:02:46.769283 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:02:46.769405 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:46.769866 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:46.770584 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:02:46.770774 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:02:46.770846 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:02:46.771040 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:02:46.771144 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:02:46.771231 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 01:02:46.771281 (+ 50us) server_negotiation.cc:299] Negotiation successful
0926 01:02:46.771300 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 01:02:46.773073 19430 raft_consensus.cc:1261] T 3ee5d737ccb14630aca97e1b007dddb9 P e71ada4ee8bf404997fb7bec1cef568f [term 1 FOLLOWER]: Refusing update from remote peer 03010feb50264e3da4b45fd1ed045323: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:02:46.773073 19701 raft_consensus.cc:1261] T 3ee5d737ccb14630aca97e1b007dddb9 P 688277c90f2f4e7d8ab35120f5b44ee5 [term 1 FOLLOWER]: Refusing update from remote peer 03010feb50264e3da4b45fd1ed045323: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:02:46.773413 19762 consensus_queue.cc:1035] T 3ee5d737ccb14630aca97e1b007dddb9 P 03010feb50264e3da4b45fd1ed045323 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e71ada4ee8bf404997fb7bec1cef568f" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 46791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:02:46.773545 19765 consensus_queue.cc:1035] T 3ee5d737ccb14630aca97e1b007dddb9 P 03010feb50264e3da4b45fd1ed045323 [LEADER]: Connected to new peer: Peer: permanent_uuid: "688277c90f2f4e7d8ab35120f5b44ee5" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:02:46.776026 19768 mvcc.cc:205] Tried to move back new op lower bound from 7333415185499918336 to 7333415185392828416. Current Snapshot: MvccSnapshot[applied={T|T < 7333415185499918336 or (T in {7333415185499918336})}]
I20260926 01:02:46.776758 19769 mvcc.cc:205] Tried to move back new op lower bound from 7333415185499918336 to 7333415185392828416. Current Snapshot: MvccSnapshot[applied={T|T < 7333415185499918336 or (T in {7333415185499918336})}]
I20260926 01:02:46.777681 19286 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:02:46.777768 19286 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:02:46.778453 19286 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4: Sending DeleteTablet for 3 replicas of tablet 3ee5d737ccb14630aca97e1b007dddb9
I20260926 01:02:46.778833 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 19325
I20260926 01:02:46.778834 19681 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ee5d737ccb14630aca97e1b007dddb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:02:46 UTC) from {username='kudu', principal='kudu/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:39032
I20260926 01:02:46.779528 19548 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ee5d737ccb14630aca97e1b007dddb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:02:46 UTC) from {username='kudu', principal='kudu/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:49524
I20260926 01:02:46.779771 19777 tablet_replica.cc:330] T 3ee5d737ccb14630aca97e1b007dddb9 P 03010feb50264e3da4b45fd1ed045323: stopping tablet replica
I20260926 01:02:46.779860 19777 raft_consensus.cc:2229] T 3ee5d737ccb14630aca97e1b007dddb9 P 03010feb50264e3da4b45fd1ed045323 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:02:46.779935 19777 raft_consensus.cc:2258] T 3ee5d737ccb14630aca97e1b007dddb9 P 03010feb50264e3da4b45fd1ed045323 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:02:46.780351 19777 ts_tablet_manager.cc:1905] T 3ee5d737ccb14630aca97e1b007dddb9 P 03010feb50264e3da4b45fd1ed045323: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:02:46.782238 19777 ts_tablet_manager.cc:1918] T 3ee5d737ccb14630aca97e1b007dddb9 P 03010feb50264e3da4b45fd1ed045323: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:02:46.782310 19777 log.cc:1196] T 3ee5d737ccb14630aca97e1b007dddb9 P 03010feb50264e3da4b45fd1ed045323: Deleting WAL directory at /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-1/wal/wals/3ee5d737ccb14630aca97e1b007dddb9
I20260926 01:02:46.782574 19777 ts_tablet_manager.cc:1939] T 3ee5d737ccb14630aca97e1b007dddb9 P 03010feb50264e3da4b45fd1ed045323: Deleting consensus metadata
I20260926 01:02:46.782979 19273 catalog_manager.cc:4841] TS 03010feb50264e3da4b45fd1ed045323 (127.3.75.2:46525): tablet 3ee5d737ccb14630aca97e1b007dddb9 (table test-table [id=1e9276e93c564b66bef6778c7959fc37]) successfully deleted
W20260926 01:02:46.784215 19273 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.75.1:46791 (error 108)
I20260926 01:02:46.785502 19776 tablet_replica.cc:330] T 3ee5d737ccb14630aca97e1b007dddb9 P 688277c90f2f4e7d8ab35120f5b44ee5: stopping tablet replica
I20260926 01:02:46.785530 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 19481
I20260926 01:02:46.785657 19776 raft_consensus.cc:2229] T 3ee5d737ccb14630aca97e1b007dddb9 P 688277c90f2f4e7d8ab35120f5b44ee5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:02:46.785753 19776 raft_consensus.cc:2258] T 3ee5d737ccb14630aca97e1b007dddb9 P 688277c90f2f4e7d8ab35120f5b44ee5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:02:46.786422 19483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:02:46.786294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.1:46791 (local address 127.0.0.1:49052)
0926 01:02:46.786351 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 01:02:46.786390 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.1:46791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 01:02:46.786592 19273 catalog_manager.cc:4584] TS e71ada4ee8bf404997fb7bec1cef568f (127.3.75.1:46791): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3ee5d737ccb14630aca97e1b007dddb9: Network error: Client connection negotiation failed: client connection to 127.3.75.1:46791: connect: Connection refused (error 111)
I20260926 01:02:46.787330 19776 ts_tablet_manager.cc:1905] T 3ee5d737ccb14630aca97e1b007dddb9 P 688277c90f2f4e7d8ab35120f5b44ee5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:02:46.789201 19776 ts_tablet_manager.cc:1918] T 3ee5d737ccb14630aca97e1b007dddb9 P 688277c90f2f4e7d8ab35120f5b44ee5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:02:46.789275 19776 log.cc:1196] T 3ee5d737ccb14630aca97e1b007dddb9 P 688277c90f2f4e7d8ab35120f5b44ee5: Deleting WAL directory at /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-2/wal/wals/3ee5d737ccb14630aca97e1b007dddb9
I20260926 01:02:46.789530 19776 ts_tablet_manager.cc:1939] T 3ee5d737ccb14630aca97e1b007dddb9 P 688277c90f2f4e7d8ab35120f5b44ee5: Deleting consensus metadata
I20260926 01:02:46.790063 19273 catalog_manager.cc:4841] TS 688277c90f2f4e7d8ab35120f5b44ee5 (127.3.75.3:46657): tablet 3ee5d737ccb14630aca97e1b007dddb9 (table test-table [id=1e9276e93c564b66bef6778c7959fc37]) successfully deleted
I20260926 01:02:46.791465 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 19618
I20260926 01:02:46.795768 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 19253
I20260926 01:03:06.801254 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:44369
--webserver_interface=127.3.75.62
--webserver_port=45483
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:41541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.75.62:44369
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:03:06.881320 19779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:03:06.881558 19779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:03:06.881593 19779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:03:06.882711 19779 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 01:03:06.882778 19779 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 01:03:06.882800 19779 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 01:03:06.882819 19779 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:03:06.882848 19779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:03:06.882905 19779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:03:06.882933 19779 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:03:06.882961 19779 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:03:06.884411 19779 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:41541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.75.62:44369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:44369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.75.62
--webserver_port=45483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:03:06.884678 19779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:03:06.884899 19779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:03:06.889598 19785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:06.889599 19787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:06.889705 19784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:03:06.889698 19779 server_base.cc:1034] running on GCE node
I20260926 01:03:06.890266 19779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:03:06.890553 19779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:03:06.891696 19779 hybrid_clock.cc:648] HybridClock initialized: now 1790384586891670 us; error 39 us; skew 500 ppm
Sep 26 01:03:06 dist-test-slave-7v10 krb5kdc[18704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384586, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:03:06.894047 19779 init.cc:377] Logged in from keytab as kudu/127.3.75.62@KRBTEST.COM (short username kudu)
I20260926 01:03:06.894510 19779 webserver.cc:460] Webserver started at http://127.3.75.62:45483/ using document root <none> and password file <none>
I20260926 01:03:06.894747 19779 fs_manager.cc:359] Metadata directory not provided
I20260926 01:03:06.894814 19779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:03:06.896191 19779 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:03:06.896855 19794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:06.897032 19779 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:03:06.897123 19779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/master-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/master-0/wal
uuid: "fb82054a1adb4eb29679b12f92a193a4"
format_stamp: "Formatted at 2026-09-26 01:02:25 on dist-test-slave-7v10"
server_key: "ea502795c473a5210aefc564fb8774c7"
server_key_iv: "a1869dc604407891798add151e72a1c4"
server_key_version: "encryptionkey@0"
I20260926 01:03:06.897464 19779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:03:06.909543 19779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:03:06.911375 19779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:03:06.911530 19779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:03:06.915493 19779 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.62:44369
I20260926 01:03:06.915539 19847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.62:44369 every 8 connection(s)
I20260926 01:03:06.915933 19779 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
I20260926 01:03:06.917313 19848 sys_catalog.cc:263] Verifying existing consensus state
I20260926 01:03:06.917946 19848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4: Bootstrap starting.
I20260926 01:03:06.926254 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 19779
I20260926 01:03:06.926281 19848 log.cc:824] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4: Log is configured to *not* fsync() on all Append() calls
I20260926 01:03:06.926601 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.1:46791
--local_ip_for_outbound_sockets=127.3.75.1
--tserver_master_addrs=127.3.75.62:44369
--webserver_port=35113
--webserver_interface=127.3.75.1
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab
--principal=kudu/127.3.75.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:41541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 01:03:06.928568 19848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 01:03:06.928886 19848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4: Bootstrap complete.
I20260926 01:03:06.931222 19848 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb82054a1adb4eb29679b12f92a193a4" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 44369 } }
I20260926 01:03:06.931465 19848 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb82054a1adb4eb29679b12f92a193a4, State: Initialized, Role: FOLLOWER
I20260926 01:03:06.931680 19848 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb82054a1adb4eb29679b12f92a193a4" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 44369 } }
I20260926 01:03:06.931779 19848 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:03:06.931833 19848 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:03:06.931881 19848 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4 [term 2 FOLLOWER]: Advancing to term 3
I20260926 01:03:06.932853 19848 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb82054a1adb4eb29679b12f92a193a4" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 44369 } }
I20260926 01:03:06.932971 19848 leader_election.cc:302] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb82054a1adb4eb29679b12f92a193a4; no voters:
I20260926 01:03:06.933167 19848 leader_election.cc:288] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 01:03:06.933202 19852 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 01:03:06.933331 19852 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4 [term 3 LEADER]: Becoming Leader. State: Replica: fb82054a1adb4eb29679b12f92a193a4, State: Running, Role: LEADER
I20260926 01:03:06.933429 19848 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:03:06.933476 19852 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb82054a1adb4eb29679b12f92a193a4" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 44369 } }
I20260926 01:03:06.933923 19854 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb82054a1adb4eb29679b12f92a193a4. Latest consensus state: current_term: 3 leader_uuid: "fb82054a1adb4eb29679b12f92a193a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb82054a1adb4eb29679b12f92a193a4" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 44369 } } }
I20260926 01:03:06.933903 19853 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "fb82054a1adb4eb29679b12f92a193a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb82054a1adb4eb29679b12f92a193a4" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 44369 } } }
I20260926 01:03:06.934051 19854 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4 [sys.catalog]: This master's current role is: LEADER
I20260926 01:03:06.934059 19853 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4 [sys.catalog]: This master's current role is: LEADER
I20260926 01:03:06.934245 19861 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:03:06.935110 19861 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:03:06.935210 19861 catalog_manager.cc:1259] Loaded cluster ID: 5454e40aff7e4a7a864d614f4a70c427
I20260926 01:03:06.935289 19861 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:03:06.936554 19861 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:03:06.936738 19861 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4: Loaded TSK: 0
I20260926 01:03:06.936789 19861 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fb82054a1adb4eb29679b12f92a193a4: Loaded TSK: 1
F20260926 01:03:06.937005 19861 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f18a1d239ad google::LogMessage::Fail() at ??:0
@ 0x7f18a1d2688a google::LogMessage::SendToLog() at ??:0
@ 0x7f18a1d234de google::LogMessage::Flush() at ??:0
@ 0x7f18a1d24a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f18a4490104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f18a448dd41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f18a9c7d1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f18a9c7d48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f18a9c7d594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f18a9c43c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f18a9c4dad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f18a3bf10d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f18a3be6954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f18a4a4a6db start_thread at ??:0
@ 0x7f18a082161f clone at ??:0
W20260926 01:03:07.040395 19851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:03:07.040599 19851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:03:07.040623 19851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:03:07.041793 19851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:03:07.041833 19851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:03:07.041859 19851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.1
I20260926 01:03:07.043360 19851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:41541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab
--principal=kudu/127.3.75.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.1:46791
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.75.1
--webserver_port=35113
--tserver_master_addrs=127.3.75.62:44369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.1
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:03:07.043602 19851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:03:07.043831 19851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:03:07.048779 19891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:07.048877 19890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:07.048781 19893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:03:07.049242 19851 server_base.cc:1034] running on GCE node
I20260926 01:03:07.049439 19851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:03:07.049717 19851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:03:07.050915 19851 hybrid_clock.cc:648] HybridClock initialized: now 1790384587050892 us; error 38 us; skew 500 ppm
Sep 26 01:03:07 dist-test-slave-7v10 krb5kdc[18704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384587, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:03:07.053579 19851 init.cc:377] Logged in from keytab as kudu/127.3.75.1@KRBTEST.COM (short username kudu)
I20260926 01:03:07.054055 19851 webserver.cc:460] Webserver started at http://127.3.75.1:35113/ using document root <none> and password file <none>
I20260926 01:03:07.054299 19851 fs_manager.cc:359] Metadata directory not provided
I20260926 01:03:07.054371 19851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:03:07.055815 19851 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:03:07.056560 19900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:07.056759 19851 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:03:07.056859 19851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-0/wal
uuid: "e71ada4ee8bf404997fb7bec1cef568f"
format_stamp: "Formatted at 2026-09-26 01:02:25 on dist-test-slave-7v10"
server_key: "f3e93385c9aac282dc820f306d26ec2d"
server_key_iv: "f5d21e0258108ead7bf184e414b60a56"
server_key_version: "encryptionkey@0"
I20260926 01:03:07.057192 19851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:03:07.075558 19851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:03:07.077844 19851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:03:07.078007 19851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:03:07.078233 19851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:03:07.078689 19907 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 01:03:07.080925 19851 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 01:03:07.081015 19851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 01:03:07.081061 19851 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 01:03:07.081707 19851 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 01:03:07.081763 19851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 01:03:07.081909 19907 tablet_bootstrap.cc:492] T 3ee5d737ccb14630aca97e1b007dddb9 P e71ada4ee8bf404997fb7bec1cef568f: Bootstrap starting.
I20260926 01:03:07.089003 19851 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.1:46791
I20260926 01:03:07.089092 20015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.1:46791 every 8 connection(s)
I20260926 01:03:07.089454 19851 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
I20260926 01:03:07.092057 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 19851
I20260926 01:03:07.092350 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.2:46525
--local_ip_for_outbound_sockets=127.3.75.2
--tserver_master_addrs=127.3.75.62:44369
--webserver_port=41721
--webserver_interface=127.3.75.2
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab
--principal=kudu/127.3.75.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:41541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 01:03:07.093971 19907 log.cc:824] T 3ee5d737ccb14630aca97e1b007dddb9 P e71ada4ee8bf404997fb7bec1cef568f: Log is configured to *not* fsync() on all Append() calls
I20260926 01:03:07.096926 19907 tablet_bootstrap.cc:492] T 3ee5d737ccb14630aca97e1b007dddb9 P e71ada4ee8bf404997fb7bec1cef568f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 01:03:07.097296 19907 tablet_bootstrap.cc:492] T 3ee5d737ccb14630aca97e1b007dddb9 P e71ada4ee8bf404997fb7bec1cef568f: Bootstrap complete.
I20260926 01:03:07.098651 19907 ts_tablet_manager.cc:1397] T 3ee5d737ccb14630aca97e1b007dddb9 P e71ada4ee8bf404997fb7bec1cef568f: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.012s
I20260926 01:03:07.100220 19907 raft_consensus.cc:348] T 3ee5d737ccb14630aca97e1b007dddb9 P e71ada4ee8bf404997fb7bec1cef568f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688277c90f2f4e7d8ab35120f5b44ee5" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46657 } } peers { permanent_uuid: "03010feb50264e3da4b45fd1ed045323" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 46525 } } peers { permanent_uuid: "e71ada4ee8bf404997fb7bec1cef568f" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 46791 } }
I20260926 01:03:07.100591 19907 raft_consensus.cc:729] T 3ee5d737ccb14630aca97e1b007dddb9 P e71ada4ee8bf404997fb7bec1cef568f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e71ada4ee8bf404997fb7bec1cef568f, State: Initialized, Role: FOLLOWER
I20260926 01:03:07.100787 19907 consensus_queue.cc:260] T 3ee5d737ccb14630aca97e1b007dddb9 P e71ada4ee8bf404997fb7bec1cef568f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688277c90f2f4e7d8ab35120f5b44ee5" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46657 } } peers { permanent_uuid: "03010feb50264e3da4b45fd1ed045323" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 46525 } } peers { permanent_uuid: "e71ada4ee8bf404997fb7bec1cef568f" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 46791 } }
I20260926 01:03:07.101078 19907 ts_tablet_manager.cc:1428] T 3ee5d737ccb14630aca97e1b007dddb9 P e71ada4ee8bf404997fb7bec1cef568f: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
W20260926 01:03:07.160269 20019 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:03:07.090128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44369 (local address 127.3.75.1:60537)
0926 01:03:07.090300 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:07.090342 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:07.090820 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:07.160073 (+ 69253us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:44369: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":96,"threads_started":1}
W20260926 01:03:07.160557 20017 heartbeater.cc:644] Failed to heartbeat to 127.3.75.62:44369 (0 consecutive failures): Network error: Failed to ping master at 127.3.75.62:44369: Client connection negotiation failed: client connection to 127.3.75.62:44369: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 01:03:07.160969 20019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:07.160807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44369 (local address 127.3.75.1:41809)
0926 01:03:07.160867 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:07.160913 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:44369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 01:03:07.161286 20019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:07.161177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44369 (local address 127.3.75.1:37355)
0926 01:03:07.161226 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:07.161256 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:44369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 01:03:07.161408 20017 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 01:03:07.176194 20020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:03:07.176383 20020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:03:07.176405 20020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:03:07.177441 20020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:03:07.177472 20020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:03:07.177496 20020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.2
I20260926 01:03:07.178802 20020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:41541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab
--principal=kudu/127.3.75.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.2:46525
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.75.2
--webserver_port=41721
--tserver_master_addrs=127.3.75.62:44369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.2
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:03:07.179029 20020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:03:07.179207 20020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:03:07.183907 20028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:07.183917 20030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:07.183947 20027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:03:07.184223 20020 server_base.cc:1034] running on GCE node
I20260926 01:03:07.184362 20020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:03:07.184525 20020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:03:07.185680 20020 hybrid_clock.cc:648] HybridClock initialized: now 1790384587185657 us; error 24 us; skew 500 ppm
Sep 26 01:03:07 dist-test-slave-7v10 krb5kdc[18704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384587, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:03:07.187999 20020 init.cc:377] Logged in from keytab as kudu/127.3.75.2@KRBTEST.COM (short username kudu)
I20260926 01:03:07.188391 20020 webserver.cc:460] Webserver started at http://127.3.75.2:41721/ using document root <none> and password file <none>
I20260926 01:03:07.188602 20020 fs_manager.cc:359] Metadata directory not provided
I20260926 01:03:07.188689 20020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:03:07.189930 20020 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:03:07.190457 20037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:07.190608 20020 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:03:07.190721 20020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-1/wal
uuid: "03010feb50264e3da4b45fd1ed045323"
format_stamp: "Formatted at 2026-09-26 01:02:25 on dist-test-slave-7v10"
server_key: "e939c6341a21860f8102d7413406ee86"
server_key_iv: "d53e19836fef09d0e066c7754aa7aa11"
server_key_version: "encryptionkey@0"
I20260926 01:03:07.191085 20020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:03:07.201628 20020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:03:07.203444 20020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:03:07.203706 20020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:03:07.203965 20020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:03:07.204262 20020 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:03:07.204316 20020 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:07.204356 20020 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:03:07.204383 20020 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:07.209975 20020 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.2:46525
I20260926 01:03:07.210069 20150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.2:46525 every 8 connection(s)
I20260926 01:03:07.210386 20020 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
I20260926 01:03:07.211447 20153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:07.211097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44369 (local address 127.3.75.2:52599)
0926 01:03:07.211249 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:07.211318 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:44369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":75,"threads_started":1}
W20260926 01:03:07.211848 20151 heartbeater.cc:644] Failed to heartbeat to 127.3.75.62:44369 (0 consecutive failures): Network error: Failed to ping master at 127.3.75.62:44369: Client connection negotiation failed: client connection to 127.3.75.62:44369: connect: Connection refused (error 111)
I20260926 01:03:07.212100 20153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:07.212016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44369 (local address 127.3.75.2:55625)
0926 01:03:07.212058 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:07.212083 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:44369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260926 01:03:07.212373 20153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:07.212300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44369 (local address 127.3.75.2:45747)
0926 01:03:07.212339 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:07.212359 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:44369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 01:03:07.212487 20151 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 01:03:07.219942 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 20020
I20260926 01:03:07.220248 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.3:46657
--local_ip_for_outbound_sockets=127.3.75.3
--tserver_master_addrs=127.3.75.62:44369
--webserver_port=34887
--webserver_interface=127.3.75.3
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab
--principal=kudu/127.3.75.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:41541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:03:07.300657 20155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:03:07.300897 20155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:03:07.300935 20155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:03:07.302064 20155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:03:07.302130 20155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:03:07.302171 20155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.3
I20260926 01:03:07.303597 20155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:41541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab
--principal=kudu/127.3.75.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.3:46657
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.75.3
--webserver_port=34887
--tserver_master_addrs=127.3.75.62:44369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.3
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:03:07.303854 20155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:03:07.304075 20155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:03:07.309505 20161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:07.309516 20160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:07.309588 20163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:03:07.309743 20155 server_base.cc:1034] running on GCE node
I20260926 01:03:07.309962 20155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:03:07.310192 20155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:03:07.311370 20155 hybrid_clock.cc:648] HybridClock initialized: now 1790384587311359 us; error 29 us; skew 500 ppm
Sep 26 01:03:07 dist-test-slave-7v10 krb5kdc[18704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384587, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:03:07.314178 20155 init.cc:377] Logged in from keytab as kudu/127.3.75.3@KRBTEST.COM (short username kudu)
I20260926 01:03:07.314625 20155 webserver.cc:460] Webserver started at http://127.3.75.3:34887/ using document root <none> and password file <none>
I20260926 01:03:07.314925 20155 fs_manager.cc:359] Metadata directory not provided
I20260926 01:03:07.315002 20155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:03:07.316258 20155 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:03:07.316887 20170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:07.317071 20155 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:03:07.317163 20155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-2/wal
uuid: "688277c90f2f4e7d8ab35120f5b44ee5"
format_stamp: "Formatted at 2026-09-26 01:02:25 on dist-test-slave-7v10"
server_key: "b50a7732bb3cb93557a9eae4cf4e2e19"
server_key_iv: "63ecaec34ec0115decafe500eff6bd3e"
server_key_version: "encryptionkey@0"
I20260926 01:03:07.317480 20155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:03:07.336771 20155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:03:07.338654 20155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:03:07.338840 20155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:03:07.339114 20155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:03:07.339476 20155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:03:07.339535 20155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:07.339582 20155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:03:07.339607 20155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:07.345414 20155 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.3:46657
I20260926 01:03:07.345455 20283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.3:46657 every 8 connection(s)
I20260926 01:03:07.345849 20155 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
I20260926 01:03:07.346984 20286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:07.346562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44369 (local address 127.3.75.3:51425)
0926 01:03:07.346716 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:07.346784 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:44369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":69,"threads_started":1}
W20260926 01:03:07.347409 20284 heartbeater.cc:644] Failed to heartbeat to 127.3.75.62:44369 (0 consecutive failures): Network error: Failed to ping master at 127.3.75.62:44369: Client connection negotiation failed: client connection to 127.3.75.62:44369: connect: Connection refused (error 111)
I20260926 01:03:07.347704 20286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:07.347606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44369 (local address 127.3.75.3:33239)
0926 01:03:07.347661 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:07.347687 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:44369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260926 01:03:07.347970 20286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:07.347899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44369 (local address 127.3.75.3:35317)
0926 01:03:07.347936 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:07.347957 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:44369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260926 01:03:07.348086 20284 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 01:03:07.355585 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 20155
I20260926 01:03:08.161907 20288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:08.161699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44369 (local address 127.3.75.1:45683)
0926 01:03:08.161814 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:08.161850 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:44369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":59,"threads_started":1}
I20260926 01:03:08.212988 20289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:08.212783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44369 (local address 127.3.75.2:49707)
0926 01:03:08.212910 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:08.212944 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:44369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":77,"threads_started":1}
I20260926 01:03:08.244743 20290 raft_consensus.cc:482] T 3ee5d737ccb14630aca97e1b007dddb9 P e71ada4ee8bf404997fb7bec1cef568f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:03:08.244872 20290 raft_consensus.cc:504] T 3ee5d737ccb14630aca97e1b007dddb9 P e71ada4ee8bf404997fb7bec1cef568f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688277c90f2f4e7d8ab35120f5b44ee5" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46657 } } peers { permanent_uuid: "03010feb50264e3da4b45fd1ed045323" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 46525 } } peers { permanent_uuid: "e71ada4ee8bf404997fb7bec1cef568f" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 46791 } }
I20260926 01:03:08.245188 20290 leader_election.cc:288] T 3ee5d737ccb14630aca97e1b007dddb9 P e71ada4ee8bf404997fb7bec1cef568f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 688277c90f2f4e7d8ab35120f5b44ee5 (127.3.75.3:46657), 03010feb50264e3da4b45fd1ed045323 (127.3.75.2:46525)
Sep 26 01:03:08 dist-test-slave-7v10 krb5kdc[18704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384587, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.1@KRBTEST.COM for kudu/127.3.75.3@KRBTEST.COM
Sep 26 01:03:08 dist-test-slave-7v10 krb5kdc[18704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384587, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.1@KRBTEST.COM for kudu/127.3.75.2@KRBTEST.COM
I20260926 01:03:08.259382 20288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:08.245372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.3:46657 (local address 127.3.75.1:39241)
0926 01:03:08.245448 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:08.245466 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:08.245556 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:08.246474 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:08.246477 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:08.246966 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:03:08.247173 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:08.247180 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:08.248303 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:08.248304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:08.249251 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:08.249253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:08.249843 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:08.249844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:08.249972 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:08.253979 (+ 4007us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:03:08.253983 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:03:08.255602 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:03:08.257617 (+ 2015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:03:08.257624 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:03:08.257626 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:03:08.257923 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:03:08.258160 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:03:08.258161 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:03:08.258162 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:03:08.258280 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:03:08.258747 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:03:08.258755 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:03:08.258925 (+ 170us) client_negotiation.cc:769] Sending connection context
0926 01:03:08.259043 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 01:03:08.259228 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":52}
I20260926 01:03:08.259464 20291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:08.245498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:39241 (local address 127.3.75.3:46657)
0926 01:03:08.245703 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:08.245717 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:08.245743 (+ 26us) server_negotiation.cc:407] Connection header received
0926 01:03:08.246198 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:08.246201 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:08.246257 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:08.246427 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:08.247302 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:08.248197 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:08.249340 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:08.249741 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:08.249868 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:08.255789 (+ 5921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:08.255794 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:08.255794 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:03:08.255809 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:08.257532 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:08.258004 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:08.258007 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:08.258007 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:08.258086 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:08.258347 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:08.258348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:08.258348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:08.258669 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:08.258756 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:08.259208 (+ 452us) server_negotiation.cc:299] Negotiation successful
0926 01:03:08.259365 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":71,"threads_started":1}
I20260926 01:03:08.259512 20292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:08.245531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.2:46525 (local address 127.3.75.1:52349)
0926 01:03:08.245668 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:08.245676 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:08.245714 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:08.246489 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:08.246490 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:08.246958 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:03:08.247175 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:08.247181 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:08.248292 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:08.248293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:08.249251 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:08.249253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:08.249845 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:08.249846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:08.249966 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:08.253931 (+ 3965us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:03:08.253940 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:03:08.256047 (+ 2107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:03:08.258195 (+ 2148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:03:08.258195 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:03:08.258196 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:03:08.258390 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:03:08.258603 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:03:08.258604 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:03:08.258604 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:03:08.258682 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:03:08.259131 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:03:08.259132 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:03:08.259169 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 01:03:08.259235 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 01:03:08.259349 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":98,"thread_start_us":78,"threads_started":1}
I20260926 01:03:08.259754 20293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:08.245658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:52349 (local address 127.3.75.2:46525)
0926 01:03:08.245791 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:08.245801 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:08.245836 (+ 35us) server_negotiation.cc:407] Connection header received
0926 01:03:08.246248 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:08.246250 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:08.246287 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:08.246446 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:08.247296 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:08.248177 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:08.249352 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:08.249753 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:08.249872 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:08.256170 (+ 6298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:08.256175 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:08.256175 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:03:08.256189 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:08.258098 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:08.258464 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:08.258467 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:08.258467 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:08.258540 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:08.258746 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:08.258747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:08.258747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:08.259035 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:08.259103 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:08.259474 (+ 371us) server_negotiation.cc:299] Negotiation successful
0926 01:03:08.259629 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":62,"threads_started":1}
W20260926 01:03:08.260524 19903 leader_election.cc:341] T 3ee5d737ccb14630aca97e1b007dddb9 P e71ada4ee8bf404997fb7bec1cef568f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 688277c90f2f4e7d8ab35120f5b44ee5 (127.3.75.3:46657): Not found: Tablet not found: 3ee5d737ccb14630aca97e1b007dddb9
W20260926 01:03:08.260624 19903 leader_election.cc:341] T 3ee5d737ccb14630aca97e1b007dddb9 P e71ada4ee8bf404997fb7bec1cef568f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 03010feb50264e3da4b45fd1ed045323 (127.3.75.2:46525): Not found: Tablet not found: 3ee5d737ccb14630aca97e1b007dddb9
I20260926 01:03:08.260660 19903 leader_election.cc:302] T 3ee5d737ccb14630aca97e1b007dddb9 P e71ada4ee8bf404997fb7bec1cef568f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e71ada4ee8bf404997fb7bec1cef568f; no voters: 03010feb50264e3da4b45fd1ed045323, 688277c90f2f4e7d8ab35120f5b44ee5
I20260926 01:03:08.260766 20290 raft_consensus.cc:2726] T 3ee5d737ccb14630aca97e1b007dddb9 P e71ada4ee8bf404997fb7bec1cef568f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 01:03:08.348536 20294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:08.348350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44369 (local address 127.3.75.3:41911)
0926 01:03:08.348462 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:08.348503 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:44369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":52,"threads_started":1}
I20260926 01:03:09.162537 20295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:09.162338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44369 (local address 127.3.75.1:34395)
0926 01:03:09.162456 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:09.162490 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:44369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":60,"threads_started":1}
I20260926 01:03:09.213869 20296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:09.213573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44369 (local address 127.3.75.2:47921)
0926 01:03:09.213794 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:09.213831 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:44369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":67,"threads_started":1}
I20260926 01:03:09.349184 20297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:09.348972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44369 (local address 127.3.75.3:49433)
0926 01:03:09.349093 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:09.349137 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:44369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":61,"threads_started":1}
I20260926 01:03:10.028290 20299 raft_consensus.cc:482] T 3ee5d737ccb14630aca97e1b007dddb9 P e71ada4ee8bf404997fb7bec1cef568f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:03:10.028425 20299 raft_consensus.cc:504] T 3ee5d737ccb14630aca97e1b007dddb9 P e71ada4ee8bf404997fb7bec1cef568f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688277c90f2f4e7d8ab35120f5b44ee5" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46657 } } peers { permanent_uuid: "03010feb50264e3da4b45fd1ed045323" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 46525 } } peers { permanent_uuid: "e71ada4ee8bf404997fb7bec1cef568f" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 46791 } }
I20260926 01:03:10.028625 20299 leader_election.cc:288] T 3ee5d737ccb14630aca97e1b007dddb9 P e71ada4ee8bf404997fb7bec1cef568f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 688277c90f2f4e7d8ab35120f5b44ee5 (127.3.75.3:46657), 03010feb50264e3da4b45fd1ed045323 (127.3.75.2:46525)
W20260926 01:03:10.029138 19903 leader_election.cc:341] T 3ee5d737ccb14630aca97e1b007dddb9 P e71ada4ee8bf404997fb7bec1cef568f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 03010feb50264e3da4b45fd1ed045323 (127.3.75.2:46525): Not found: Tablet not found: 3ee5d737ccb14630aca97e1b007dddb9
W20260926 01:03:10.029232 19903 leader_election.cc:341] T 3ee5d737ccb14630aca97e1b007dddb9 P e71ada4ee8bf404997fb7bec1cef568f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 688277c90f2f4e7d8ab35120f5b44ee5 (127.3.75.3:46657): Not found: Tablet not found: 3ee5d737ccb14630aca97e1b007dddb9
I20260926 01:03:10.029249 19903 leader_election.cc:302] T 3ee5d737ccb14630aca97e1b007dddb9 P e71ada4ee8bf404997fb7bec1cef568f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e71ada4ee8bf404997fb7bec1cef568f; no voters: 03010feb50264e3da4b45fd1ed045323, 688277c90f2f4e7d8ab35120f5b44ee5
I20260926 01:03:10.029366 20299 raft_consensus.cc:2726] T 3ee5d737ccb14630aca97e1b007dddb9 P e71ada4ee8bf404997fb7bec1cef568f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 01:03:10.163312 20300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:10.163018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44369 (local address 127.3.75.1:42573)
0926 01:03:10.163216 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:10.163263 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:44369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":64,"threads_started":1}
I20260926 01:03:10.214568 20301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:10.214341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44369 (local address 127.3.75.2:59229)
0926 01:03:10.214480 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:10.214526 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:44369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":63,"threads_started":1}
I20260926 01:03:10.349879 20302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:10.349646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44369 (local address 127.3.75.3:50943)
0926 01:03:10.349789 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:10.349836 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:44369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":79,"threads_started":1}
I20260926 01:03:11.163955 20303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:11.163743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44369 (local address 127.3.75.1:39829)
0926 01:03:11.163870 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:11.163911 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:44369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":70,"threads_started":1}
I20260926 01:03:11.215412 20304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:11.215189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44369 (local address 127.3.75.2:43511)
0926 01:03:11.215347 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:11.215380 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:44369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":96,"threads_started":1}
I20260926 01:03:11.350538 20305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:11.350329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44369 (local address 127.3.75.3:41999)
0926 01:03:11.350461 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:11.350499 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:44369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":72,"threads_started":1}
I20260926 01:03:12.164633 20306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:12.164428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44369 (local address 127.3.75.1:41037)
0926 01:03:12.164549 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:12.164585 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:44369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":63,"threads_started":1}
I20260926 01:03:12.215994 20307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:12.215795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44369 (local address 127.3.75.2:49855)
0926 01:03:12.215922 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:12.215958 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:44369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":70,"threads_started":1}
I20260926 01:03:12.351289 20308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:12.351045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44369 (local address 127.3.75.3:54679)
0926 01:03:12.351191 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:12.351234 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:44369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":70,"threads_started":1}
I20260926 01:03:13.165381 20309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:13.165173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44369 (local address 127.3.75.1:32931)
0926 01:03:13.165298 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:13.165337 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:44369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":67,"threads_started":1}
I20260926 01:03:13.191825 20310 raft_consensus.cc:482] T 3ee5d737ccb14630aca97e1b007dddb9 P e71ada4ee8bf404997fb7bec1cef568f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:03:13.191916 20310 raft_consensus.cc:504] T 3ee5d737ccb14630aca97e1b007dddb9 P e71ada4ee8bf404997fb7bec1cef568f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688277c90f2f4e7d8ab35120f5b44ee5" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46657 } } peers { permanent_uuid: "03010feb50264e3da4b45fd1ed045323" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 46525 } } peers { permanent_uuid: "e71ada4ee8bf404997fb7bec1cef568f" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 46791 } }
I20260926 01:03:13.192039 20310 leader_election.cc:288] T 3ee5d737ccb14630aca97e1b007dddb9 P e71ada4ee8bf404997fb7bec1cef568f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 688277c90f2f4e7d8ab35120f5b44ee5 (127.3.75.3:46657), 03010feb50264e3da4b45fd1ed045323 (127.3.75.2:46525)
W20260926 01:03:13.192417 19903 leader_election.cc:341] T 3ee5d737ccb14630aca97e1b007dddb9 P e71ada4ee8bf404997fb7bec1cef568f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 688277c90f2f4e7d8ab35120f5b44ee5 (127.3.75.3:46657): Not found: Tablet not found: 3ee5d737ccb14630aca97e1b007dddb9
W20260926 01:03:13.192500 19903 leader_election.cc:341] T 3ee5d737ccb14630aca97e1b007dddb9 P e71ada4ee8bf404997fb7bec1cef568f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 03010feb50264e3da4b45fd1ed045323 (127.3.75.2:46525): Not found: Tablet not found: 3ee5d737ccb14630aca97e1b007dddb9
I20260926 01:03:13.192545 19903 leader_election.cc:302] T 3ee5d737ccb14630aca97e1b007dddb9 P e71ada4ee8bf404997fb7bec1cef568f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e71ada4ee8bf404997fb7bec1cef568f; no voters: 03010feb50264e3da4b45fd1ed045323, 688277c90f2f4e7d8ab35120f5b44ee5
I20260926 01:03:13.192633 20310 raft_consensus.cc:2726] T 3ee5d737ccb14630aca97e1b007dddb9 P e71ada4ee8bf404997fb7bec1cef568f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 01:03:13.216661 20311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:13.216439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44369 (local address 127.3.75.2:56777)
0926 01:03:13.216581 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:13.216621 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:44369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":58,"threads_started":1}
I20260926 01:03:13.351950 20312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:13.351754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44369 (local address 127.3.75.3:43609)
0926 01:03:13.351879 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:13.351914 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:44369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":58,"threads_started":1}
I20260926 01:03:14.166044 20313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:14.165831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44369 (local address 127.3.75.1:34291)
0926 01:03:14.165961 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:14.165998 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:44369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":67,"threads_started":1}
I20260926 01:03:14.217262 20314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:14.217082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44369 (local address 127.3.75.2:60975)
0926 01:03:14.217193 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:14.217225 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:44369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":56,"threads_started":1}
I20260926 01:03:14.352713 20315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:14.352461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44369 (local address 127.3.75.3:54363)
0926 01:03:14.352609 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:14.352662 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:44369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":73,"threads_started":1}
I20260926 01:03:15.166746 20316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:15.166512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44369 (local address 127.3.75.1:36109)
0926 01:03:15.166641 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:15.166700 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:44369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":63,"threads_started":1}
I20260926 01:03:15.217927 20317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:15.217736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44369 (local address 127.3.75.2:47741)
0926 01:03:15.217854 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:15.217888 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:44369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":61,"threads_started":1}
I20260926 01:03:15.353443 20318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:15.353246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44369 (local address 127.3.75.3:43045)
0926 01:03:15.353367 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:15.353402 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:44369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":63,"threads_started":1}
I20260926 01:03:16.167444 20319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:16.167211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44369 (local address 127.3.75.1:50521)
0926 01:03:16.167348 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:16.167394 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:44369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":64,"threads_started":1}
I20260926 01:03:16.218585 20320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:16.218378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44369 (local address 127.3.75.2:39695)
0926 01:03:16.218505 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:16.218543 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:44369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":63,"threads_started":1}
I20260926 01:03:16.354141 20321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:16.353890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44369 (local address 127.3.75.3:57871)
0926 01:03:16.354064 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:16.354099 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:44369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":109,"threads_started":1}
I20260926 01:03:17.168177 20322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:17.167911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44369 (local address 127.3.75.1:56793)
0926 01:03:17.168095 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:17.168130 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:44369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":88,"threads_started":1}
I20260926 01:03:17.219189 20323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:17.218993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44369 (local address 127.3.75.2:44975)
0926 01:03:17.219109 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:17.219140 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:44369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":57,"threads_started":1}
I20260926 01:03:17.354743 20324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:17.354555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44369 (local address 127.3.75.3:47851)
0926 01:03:17.354669 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:17.354699 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:44369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":58,"threads_started":1}
I20260926 01:03:17.797338 20325 raft_consensus.cc:482] T 3ee5d737ccb14630aca97e1b007dddb9 P e71ada4ee8bf404997fb7bec1cef568f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:03:17.797442 20325 raft_consensus.cc:504] T 3ee5d737ccb14630aca97e1b007dddb9 P e71ada4ee8bf404997fb7bec1cef568f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688277c90f2f4e7d8ab35120f5b44ee5" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46657 } } peers { permanent_uuid: "03010feb50264e3da4b45fd1ed045323" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 46525 } } peers { permanent_uuid: "e71ada4ee8bf404997fb7bec1cef568f" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 46791 } }
I20260926 01:03:17.797596 20325 leader_election.cc:288] T 3ee5d737ccb14630aca97e1b007dddb9 P e71ada4ee8bf404997fb7bec1cef568f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 688277c90f2f4e7d8ab35120f5b44ee5 (127.3.75.3:46657), 03010feb50264e3da4b45fd1ed045323 (127.3.75.2:46525)
W20260926 01:03:17.797982 19903 leader_election.cc:341] T 3ee5d737ccb14630aca97e1b007dddb9 P e71ada4ee8bf404997fb7bec1cef568f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 688277c90f2f4e7d8ab35120f5b44ee5 (127.3.75.3:46657): Not found: Tablet not found: 3ee5d737ccb14630aca97e1b007dddb9
W20260926 01:03:17.798053 19903 leader_election.cc:341] T 3ee5d737ccb14630aca97e1b007dddb9 P e71ada4ee8bf404997fb7bec1cef568f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 03010feb50264e3da4b45fd1ed045323 (127.3.75.2:46525): Not found: Tablet not found: 3ee5d737ccb14630aca97e1b007dddb9
I20260926 01:03:17.798069 19903 leader_election.cc:302] T 3ee5d737ccb14630aca97e1b007dddb9 P e71ada4ee8bf404997fb7bec1cef568f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e71ada4ee8bf404997fb7bec1cef568f; no voters: 03010feb50264e3da4b45fd1ed045323, 688277c90f2f4e7d8ab35120f5b44ee5
I20260926 01:03:17.798138 20325 raft_consensus.cc:2726] T 3ee5d737ccb14630aca97e1b007dddb9 P e71ada4ee8bf404997fb7bec1cef568f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 01:03:18.168905 20326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:18.168684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44369 (local address 127.3.75.1:60889)
0926 01:03:18.168820 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:18.168858 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:44369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":61,"threads_started":1}
I20260926 01:03:18.219941 20327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:18.219726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44369 (local address 127.3.75.2:49981)
0926 01:03:18.219849 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:18.219883 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:44369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":61,"threads_started":1}
I20260926 01:03:18.355398 20328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:18.355200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44369 (local address 127.3.75.3:37807)
0926 01:03:18.355331 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:18.355362 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:44369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":73,"threads_started":1}
I20260926 01:03:19.169752 20329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:19.169478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44369 (local address 127.3.75.1:37875)
0926 01:03:19.169665 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:19.169706 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:44369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":66,"threads_started":1}
I20260926 01:03:19.220638 20330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:19.220371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44369 (local address 127.3.75.2:54087)
0926 01:03:19.220521 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:19.220570 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:44369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":67,"threads_started":1}
I20260926 01:03:19.355989 20331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:19.355793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44369 (local address 127.3.75.3:58859)
0926 01:03:19.355914 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:19.355949 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:44369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":59,"threads_started":1}
I20260926 01:03:20.170542 20332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:20.170346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44369 (local address 127.3.75.1:55495)
0926 01:03:20.170463 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:20.170497 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:44369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":63,"threads_started":1}
I20260926 01:03:20.221266 20333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:20.221058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44369 (local address 127.3.75.2:55623)
0926 01:03:20.221188 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:20.221223 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:44369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":56,"threads_started":1}
I20260926 01:03:20.356590 20334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:20.356390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44369 (local address 127.3.75.3:54393)
0926 01:03:20.356510 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:20.356550 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:44369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":58,"threads_started":1}
I20260926 01:03:21.171450 20335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:21.171165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44369 (local address 127.3.75.1:53029)
0926 01:03:21.171341 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:21.171380 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:44369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":68,"threads_started":1}
I20260926 01:03:21.221885 20336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:21.221680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44369 (local address 127.3.75.2:47135)
0926 01:03:21.221801 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:21.221839 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:44369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":58,"threads_started":1}
I20260926 01:03:21.357254 20337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:21.357032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44369 (local address 127.3.75.3:56867)
0926 01:03:21.357152 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:21.357210 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:44369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":61,"threads_started":1}
I20260926 01:03:21.369697 20338 raft_consensus.cc:482] T 3ee5d737ccb14630aca97e1b007dddb9 P e71ada4ee8bf404997fb7bec1cef568f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:03:21.369830 20338 raft_consensus.cc:504] T 3ee5d737ccb14630aca97e1b007dddb9 P e71ada4ee8bf404997fb7bec1cef568f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688277c90f2f4e7d8ab35120f5b44ee5" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 46657 } } peers { permanent_uuid: "03010feb50264e3da4b45fd1ed045323" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 46525 } } peers { permanent_uuid: "e71ada4ee8bf404997fb7bec1cef568f" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 46791 } }
I20260926 01:03:21.370011 20338 leader_election.cc:288] T 3ee5d737ccb14630aca97e1b007dddb9 P e71ada4ee8bf404997fb7bec1cef568f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 688277c90f2f4e7d8ab35120f5b44ee5 (127.3.75.3:46657), 03010feb50264e3da4b45fd1ed045323 (127.3.75.2:46525)
W20260926 01:03:21.370431 19903 leader_election.cc:341] T 3ee5d737ccb14630aca97e1b007dddb9 P e71ada4ee8bf404997fb7bec1cef568f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 03010feb50264e3da4b45fd1ed045323 (127.3.75.2:46525): Not found: Tablet not found: 3ee5d737ccb14630aca97e1b007dddb9
W20260926 01:03:21.370496 19903 leader_election.cc:341] T 3ee5d737ccb14630aca97e1b007dddb9 P e71ada4ee8bf404997fb7bec1cef568f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 688277c90f2f4e7d8ab35120f5b44ee5 (127.3.75.3:46657): Not found: Tablet not found: 3ee5d737ccb14630aca97e1b007dddb9
I20260926 01:03:21.370512 19903 leader_election.cc:302] T 3ee5d737ccb14630aca97e1b007dddb9 P e71ada4ee8bf404997fb7bec1cef568f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e71ada4ee8bf404997fb7bec1cef568f; no voters: 03010feb50264e3da4b45fd1ed045323, 688277c90f2f4e7d8ab35120f5b44ee5
I20260926 01:03:21.370585 20338 raft_consensus.cc:2726] T 3ee5d737ccb14630aca97e1b007dddb9 P e71ada4ee8bf404997fb7bec1cef568f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 01:03:22.172106 20339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:22.171909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44369 (local address 127.3.75.1:44187)
0926 01:03:22.172031 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:22.172062 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:44369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":71,"threads_started":1}
I20260926 01:03:22.222692 20340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:22.222477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44369 (local address 127.3.75.2:50781)
0926 01:03:22.222605 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:22.222637 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:44369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":74,"threads_started":1}
I20260926 01:03:22.357913 20341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:22.357712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44369 (local address 127.3.75.3:43827)
0926 01:03:22.357842 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:22.357872 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.62:44369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":78,"threads_started":1}
I20260926 01:03:22.365507 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 19851
I20260926 01:03:22.370592 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 20020
I20260926 01:03:22.375021 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 20155
2026-09-26T01:03:22Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58500 ms)
[----------] 30 tests from SecurityITest (230308 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:03:22 dist-test-slave-7v10 krb5kdc[20344](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:03:22 dist-test-slave-7v10 krb5kdc[20344](info): set up 2 sockets
Sep 26 01:03:22 dist-test-slave-7v10 krb5kdc[20344](info): commencing operation
krb5kdc: starting...
W20260926 01:03:23.417869 3372 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Sep 26 01:03:23 dist-test-slave-7v10 krb5kdc[20344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:03:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:03:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.75.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab.
Entry for principal kudu/127.3.75.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab.
I20260926 01:03:23.533957 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:38481
--webserver_interface=127.3.75.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:41297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.75.62:38481
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:03:23.612862 20357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:03:23.613057 20357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:03:23.613078 20357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:03:23.614240 20357 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:03:23.614282 20357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:03:23.614297 20357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:03:23.614310 20357 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:03:23.614322 20357 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:03:23.615840 20357 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:41297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.75.62:38481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:38481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:03:23.616081 20357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:03:23.616326 20357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:03:23.621275 20362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:23.621313 20365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:03:23.621397 20357 server_base.cc:1034] running on GCE node
W20260926 01:03:23.621356 20363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:03:23.621726 20357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:03:23.622013 20357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:03:23.623149 20357 hybrid_clock.cc:648] HybridClock initialized: now 1790384603623130 us; error 35 us; skew 500 ppm
Sep 26 01:03:23 dist-test-slave-7v10 krb5kdc[20344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384603, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:03:23.625437 20357 init.cc:377] Logged in from keytab as kudu/127.3.75.62@KRBTEST.COM (short username kudu)
I20260926 01:03:23.625908 20357 webserver.cc:460] Webserver started at http://127.3.75.62:33881/ using document root <none> and password file <none>
I20260926 01:03:23.626102 20357 fs_manager.cc:359] Metadata directory not provided
I20260926 01:03:23.626140 20357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:03:23.626230 20357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:03:23.627256 20357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/master-0/data/instance:
uuid: "8088e19ce2b940f1bf94d8200c557963"
format_stamp: "Formatted at 2026-09-26 01:03:23 on dist-test-slave-7v10"
server_key: "9ad85fd5937f60c5c62a7317ded779e5"
server_key_iv: "8c972f0800f744e7d9e691efb85e5bb7"
server_key_version: "encryptionkey@0"
I20260926 01:03:23.627542 20357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/master-0/wal/instance:
uuid: "8088e19ce2b940f1bf94d8200c557963"
format_stamp: "Formatted at 2026-09-26 01:03:23 on dist-test-slave-7v10"
server_key: "9ad85fd5937f60c5c62a7317ded779e5"
server_key_iv: "8c972f0800f744e7d9e691efb85e5bb7"
server_key_version: "encryptionkey@0"
I20260926 01:03:23.628737 20357 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:03:23.629417 20372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:23.629598 20357 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:03:23.629667 20357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/master-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/master-0/wal
uuid: "8088e19ce2b940f1bf94d8200c557963"
format_stamp: "Formatted at 2026-09-26 01:03:23 on dist-test-slave-7v10"
server_key: "9ad85fd5937f60c5c62a7317ded779e5"
server_key_iv: "8c972f0800f744e7d9e691efb85e5bb7"
server_key_version: "encryptionkey@0"
I20260926 01:03:23.629716 20357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:03:23.654072 20357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:03:23.656044 20357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:03:23.656157 20357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:03:23.660471 20357 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.62:38481
I20260926 01:03:23.660517 20424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.62:38481 every 8 connection(s)
I20260926 01:03:23.660923 20357 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
I20260926 01:03:23.661785 20425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:03:23.664031 20425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8088e19ce2b940f1bf94d8200c557963: Bootstrap starting.
I20260926 01:03:23.664570 20425 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8088e19ce2b940f1bf94d8200c557963: Neither blocks nor log segments found. Creating new log.
I20260926 01:03:23.664804 20425 log.cc:824] T 00000000000000000000000000000000 P 8088e19ce2b940f1bf94d8200c557963: Log is configured to *not* fsync() on all Append() calls
I20260926 01:03:23.665432 20425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8088e19ce2b940f1bf94d8200c557963: No bootstrap required, opened a new log
I20260926 01:03:23.667029 20425 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8088e19ce2b940f1bf94d8200c557963 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8088e19ce2b940f1bf94d8200c557963" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 38481 } }
I20260926 01:03:23.667164 20425 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8088e19ce2b940f1bf94d8200c557963 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:03:23.667187 20425 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8088e19ce2b940f1bf94d8200c557963 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8088e19ce2b940f1bf94d8200c557963, State: Initialized, Role: FOLLOWER
I20260926 01:03:23.667316 20425 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8088e19ce2b940f1bf94d8200c557963 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8088e19ce2b940f1bf94d8200c557963" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 38481 } }
I20260926 01:03:23.667393 20425 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8088e19ce2b940f1bf94d8200c557963 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:03:23.667415 20425 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8088e19ce2b940f1bf94d8200c557963 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:03:23.667444 20425 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8088e19ce2b940f1bf94d8200c557963 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:03:23.667919 20425 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8088e19ce2b940f1bf94d8200c557963 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8088e19ce2b940f1bf94d8200c557963" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 38481 } }
I20260926 01:03:23.667997 20425 leader_election.cc:302] T 00000000000000000000000000000000 P 8088e19ce2b940f1bf94d8200c557963 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8088e19ce2b940f1bf94d8200c557963; no voters:
I20260926 01:03:23.668118 20425 leader_election.cc:288] T 00000000000000000000000000000000 P 8088e19ce2b940f1bf94d8200c557963 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:03:23.668192 20428 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8088e19ce2b940f1bf94d8200c557963 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:03:23.668370 20425 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8088e19ce2b940f1bf94d8200c557963 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:03:23.668397 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 20357
I20260926 01:03:23.668422 20428 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8088e19ce2b940f1bf94d8200c557963 [term 1 LEADER]: Becoming Leader. State: Replica: 8088e19ce2b940f1bf94d8200c557963, State: Running, Role: LEADER
I20260926 01:03:23.668506 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/master-0/wal/instance
I20260926 01:03:23.668516 20428 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8088e19ce2b940f1bf94d8200c557963 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8088e19ce2b940f1bf94d8200c557963" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 38481 } }
I20260926 01:03:23.668701 3372 external_mini_cluster.cc:1428] Setting key b0f275ffb9554aefec00593df4fd53cf
I20260926 01:03:23.668931 20430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8088e19ce2b940f1bf94d8200c557963 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8088e19ce2b940f1bf94d8200c557963. Latest consensus state: current_term: 1 leader_uuid: "8088e19ce2b940f1bf94d8200c557963" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8088e19ce2b940f1bf94d8200c557963" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 38481 } } }
I20260926 01:03:23.668998 20429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8088e19ce2b940f1bf94d8200c557963 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8088e19ce2b940f1bf94d8200c557963" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8088e19ce2b940f1bf94d8200c557963" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 38481 } } }
I20260926 01:03:23.669234 20429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8088e19ce2b940f1bf94d8200c557963 [sys.catalog]: This master's current role is: LEADER
I20260926 01:03:23.669461 20430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8088e19ce2b940f1bf94d8200c557963 [sys.catalog]: This master's current role is: LEADER
I20260926 01:03:23.669762 20440 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:03:23.670440 20440 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:03:23.672003 20440 catalog_manager.cc:1347] Generated new cluster ID: 10b3f159d08f4e3ba13f5309d2ef2c58
I20260926 01:03:23.672058 20440 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 01:03:23 dist-test-slave-7v10 krb5kdc[20344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 01:03:23.680281 20443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:23.669435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52956 (local address 127.3.75.62:38481)
0926 01:03:23.669868 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:23.669878 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:23.669911 (+ 33us) server_negotiation.cc:407] Connection header received
0926 01:03:23.670027 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:23.670034 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:23.670100 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:23.670242 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:23.670987 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:23.672273 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:23.673083 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:23.673530 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:23.673669 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:23.676240 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:23.676244 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:23.676245 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:03:23.676257 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:23.678594 (+ 2337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:23.679027 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:23.679030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:23.679031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:23.679142 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:23.679382 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:23.679383 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:23.679383 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:23.679627 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:23.679808 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:23.679912 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 01:03:23.680057 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":81,"threads_started":1}
I20260926 01:03:23.680792 20440 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:03:23.681392 20440 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:03:23.686194 20440 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8088e19ce2b940f1bf94d8200c557963: Generated new TSK 0
I20260926 01:03:23.686357 20440 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.75.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab.
Entry for principal kudu/127.3.75.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab.
I20260926 01:03:23.714458 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.1:0
--local_ip_for_outbound_sockets=127.3.75.1
--webserver_interface=127.3.75.1
--webserver_port=0
--tserver_master_addrs=127.3.75.62:38481
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab
--principal=kudu/127.3.75.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:41297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:03:23.795639 20452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:03:23.795823 20452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:03:23.795846 20452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:03:23.797008 20452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:03:23.797056 20452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:03:23.797083 20452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.1
I20260926 01:03:23.798503 20452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:41297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab
--principal=kudu/127.3.75.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.75.1
--webserver_port=0
--tserver_master_addrs=127.3.75.62:38481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.1
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:03:23.798758 20452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:03:23.799043 20452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:03:23.804183 20458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:23.804188 20457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:23.804241 20460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:03:23.804711 20452 server_base.cc:1034] running on GCE node
I20260926 01:03:23.804946 20452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:03:23.805212 20452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:03:23.806367 20452 hybrid_clock.cc:648] HybridClock initialized: now 1790384603806357 us; error 26 us; skew 500 ppm
Sep 26 01:03:23 dist-test-slave-7v10 krb5kdc[20344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384603, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:03:23.808813 20452 init.cc:377] Logged in from keytab as kudu/127.3.75.1@KRBTEST.COM (short username kudu)
I20260926 01:03:23.809302 20452 webserver.cc:460] Webserver started at http://127.3.75.1:39461/ using document root <none> and password file <none>
I20260926 01:03:23.809590 20452 fs_manager.cc:359] Metadata directory not provided
I20260926 01:03:23.809669 20452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:03:23.809805 20452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:03:23.811071 20452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/ts-0/data/instance:
uuid: "39e8d4a39a604a68b0f2a8aee0bd848b"
format_stamp: "Formatted at 2026-09-26 01:03:23 on dist-test-slave-7v10"
server_key: "36fd76719a1997507a5d6f052fe3755d"
server_key_iv: "10c2c953c911245fef0a0fcbc3bfd56a"
server_key_version: "encryptionkey@0"
I20260926 01:03:23.811453 20452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/ts-0/wal/instance:
uuid: "39e8d4a39a604a68b0f2a8aee0bd848b"
format_stamp: "Formatted at 2026-09-26 01:03:23 on dist-test-slave-7v10"
server_key: "36fd76719a1997507a5d6f052fe3755d"
server_key_iv: "10c2c953c911245fef0a0fcbc3bfd56a"
server_key_version: "encryptionkey@0"
I20260926 01:03:23.812750 20452 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:03:23.813494 20467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:23.813664 20452 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:03:23.813766 20452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/ts-0/wal
uuid: "39e8d4a39a604a68b0f2a8aee0bd848b"
format_stamp: "Formatted at 2026-09-26 01:03:23 on dist-test-slave-7v10"
server_key: "36fd76719a1997507a5d6f052fe3755d"
server_key_iv: "10c2c953c911245fef0a0fcbc3bfd56a"
server_key_version: "encryptionkey@0"
I20260926 01:03:23.813841 20452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:03:23.825767 20452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:03:23.828020 20452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:03:23.828179 20452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:03:23.828408 20452 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:03:23.828696 20452 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:03:23.828729 20452 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:23.828750 20452 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:03:23.828796 20452 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:23.834744 20452 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.1:42727
I20260926 01:03:23.834805 20580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.1:42727 every 8 connection(s)
I20260926 01:03:23.835103 20452 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
I20260926 01:03:23.839084 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 20452
I20260926 01:03:23.839193 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/ts-0/wal/instance
I20260926 01:03:23.839409 3372 external_mini_cluster.cc:1428] Setting key 1cd75c5bb033bd7a5077452f05c95f77
Sep 26 01:03:23 dist-test-slave-7v10 krb5kdc[20344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384603, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.1@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 01:03:23.844578 20443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:23.835810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:49679 (local address 127.3.75.62:38481)
0926 01:03:23.835904 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:23.835906 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:23.836116 (+ 210us) server_negotiation.cc:407] Connection header received
0926 01:03:23.836582 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:23.836583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:23.836592 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:23.836649 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:23.837497 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:23.837985 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:23.838977 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:23.839340 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:23.839419 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:23.841663 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:23.841665 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:23.841665 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:03:23.841670 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:23.843145 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:23.843648 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:23.843648 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:23.843649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:23.843712 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:23.843976 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:23.843977 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:23.843977 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:23.844070 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:23.844117 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:23.844456 (+ 339us) server_negotiation.cc:299] Negotiation successful
0926 01:03:23.844517 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 01:03:23.844879 20583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:23.835805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:38481 (local address 127.3.75.1:49679)
0926 01:03:23.835994 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:23.836033 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:23.836503 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:23.836700 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:23.836703 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:23.837235 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:03:23.837421 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:23.837426 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:23.838062 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:23.838063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:23.838834 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:23.838836 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:23.839426 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:23.839427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:23.839562 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:23.840110 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:03:23.840118 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:03:23.841567 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:03:23.843226 (+ 1659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:03:23.843233 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:03:23.843235 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:03:23.843568 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:03:23.843777 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:03:23.843778 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:03:23.843779 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:03:23.843911 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:03:23.844134 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:03:23.844140 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:03:23.844303 (+ 163us) client_negotiation.cc:769] Sending connection context
0926 01:03:23.844427 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 01:03:23.844599 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":124,"threads_started":1}
I20260926 01:03:23.845446 20581 heartbeater.cc:347] Connected to a master server at 127.3.75.62:38481
I20260926 01:03:23.845542 20581 heartbeater.cc:464] Registering TS with master...
I20260926 01:03:23.845736 20581 heartbeater.cc:511] Master 127.3.75.62:38481 requested a full tablet report, sending...
I20260926 01:03:23.846249 20389 ts_manager.cc:194] Registered new tserver with Master: 39e8d4a39a604a68b0f2a8aee0bd848b (127.3.75.1:42727)
I20260926 01:03:23.846849 20389 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.75.1@KRBTEST.COM'} at 127.3.75.1:49679
WARNING: no policy specified for kudu/127.3.75.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab.
Entry for principal kudu/127.3.75.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab.
I20260926 01:03:23.863871 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.2:0
--local_ip_for_outbound_sockets=127.3.75.2
--webserver_interface=127.3.75.2
--webserver_port=0
--tserver_master_addrs=127.3.75.62:38481
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab
--principal=kudu/127.3.75.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:41297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:03:23.948164 20586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:03:23.948413 20586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:03:23.948453 20586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:03:23.949636 20586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:03:23.949708 20586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:03:23.949757 20586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.2
I20260926 01:03:23.951344 20586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:41297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab
--principal=kudu/127.3.75.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.75.2
--webserver_port=0
--tserver_master_addrs=127.3.75.62:38481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.2
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:03:23.951596 20586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:03:23.951834 20586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:03:23.956619 20592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:23.956629 20594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:23.956700 20591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:03:23.957047 20586 server_base.cc:1034] running on GCE node
I20260926 01:03:23.957245 20586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:03:23.957484 20586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:03:23.958750 20586 hybrid_clock.cc:648] HybridClock initialized: now 1790384603958724 us; error 52 us; skew 500 ppm
Sep 26 01:03:23 dist-test-slave-7v10 krb5kdc[20344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384603, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:03:23.961189 20586 init.cc:377] Logged in from keytab as kudu/127.3.75.2@KRBTEST.COM (short username kudu)
I20260926 01:03:23.961654 20586 webserver.cc:460] Webserver started at http://127.3.75.2:44505/ using document root <none> and password file <none>
I20260926 01:03:23.961891 20586 fs_manager.cc:359] Metadata directory not provided
I20260926 01:03:23.961959 20586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:03:23.962087 20586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:03:23.963163 20586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/ts-1/data/instance:
uuid: "c4d2a51a02b7433ebcff87c573ccc733"
format_stamp: "Formatted at 2026-09-26 01:03:23 on dist-test-slave-7v10"
server_key: "2f71f9696ff9fe3dad2cd3f294d374ab"
server_key_iv: "1579ab022d89e16695cb74e534fb7aca"
server_key_version: "encryptionkey@0"
I20260926 01:03:23.963513 20586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/ts-1/wal/instance:
uuid: "c4d2a51a02b7433ebcff87c573ccc733"
format_stamp: "Formatted at 2026-09-26 01:03:23 on dist-test-slave-7v10"
server_key: "2f71f9696ff9fe3dad2cd3f294d374ab"
server_key_iv: "1579ab022d89e16695cb74e534fb7aca"
server_key_version: "encryptionkey@0"
I20260926 01:03:23.964771 20586 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:03:23.965463 20601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:23.965623 20586 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:03:23.965727 20586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/ts-1/wal
uuid: "c4d2a51a02b7433ebcff87c573ccc733"
format_stamp: "Formatted at 2026-09-26 01:03:23 on dist-test-slave-7v10"
server_key: "2f71f9696ff9fe3dad2cd3f294d374ab"
server_key_iv: "1579ab022d89e16695cb74e534fb7aca"
server_key_version: "encryptionkey@0"
I20260926 01:03:23.965807 20586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:03:23.980086 20586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:03:23.981797 20586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:03:23.981946 20586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:03:23.982152 20586 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:03:23.982471 20586 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:03:23.982524 20586 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:23.982563 20586 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:03:23.982590 20586 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:23.988258 20586 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.2:39189
I20260926 01:03:23.988303 20716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.2:39189 every 8 connection(s)
I20260926 01:03:23.988580 20586 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
Sep 26 01:03:23 dist-test-slave-7v10 krb5kdc[20344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384603, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.2@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 01:03:23.996706 20443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:23.989283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:40383 (local address 127.3.75.62:38481)
0926 01:03:23.989330 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:23.989332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:23.989513 (+ 181us) server_negotiation.cc:407] Connection header received
0926 01:03:23.989962 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:23.989963 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:23.989972 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:23.990023 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:23.990842 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:23.991318 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:23.992205 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:23.992377 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:23.992426 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:23.994330 (+ 1904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:23.994332 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:23.994332 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:03:23.994338 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:23.995557 (+ 1219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:23.995977 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:23.995977 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:23.995978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:23.996026 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:23.996237 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:23.996238 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:23.996238 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:23.996327 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:23.996373 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:23.996578 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 01:03:23.996637 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 01:03:23.996874 20719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:23.989229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:38481 (local address 127.3.75.2:40383)
0926 01:03:23.989385 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:23.989424 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:23.989884 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:23.990070 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:23.990074 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:23.990592 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:03:23.990769 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:23.990773 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:23.991394 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:23.991395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:23.992128 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:23.992129 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:23.992445 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:23.992446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:23.992537 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:23.992934 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:03:23.992941 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:03:23.994249 (+ 1308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:03:23.995662 (+ 1413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:03:23.995669 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:03:23.995670 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:03:23.995897 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:03:23.996082 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:03:23.996083 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:03:23.996083 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:03:23.996188 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:03:23.996378 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:03:23.996382 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:03:23.996480 (+ 98us) client_negotiation.cc:769] Sending connection context
0926 01:03:23.996553 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 01:03:23.996656 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":90,"threads_started":1}
I20260926 01:03:23.997407 20717 heartbeater.cc:347] Connected to a master server at 127.3.75.62:38481
I20260926 01:03:23.997483 20717 heartbeater.cc:464] Registering TS with master...
I20260926 01:03:23.997685 20717 heartbeater.cc:511] Master 127.3.75.62:38481 requested a full tablet report, sending...
I20260926 01:03:23.998046 20389 ts_manager.cc:194] Registered new tserver with Master: c4d2a51a02b7433ebcff87c573ccc733 (127.3.75.2:39189)
I20260926 01:03:23.998343 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 20586
I20260926 01:03:23.998445 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/ts-1/wal/instance
I20260926 01:03:23.998554 20389 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.75.2@KRBTEST.COM'} at 127.3.75.2:40383
I20260926 01:03:23.998639 3372 external_mini_cluster.cc:1428] Setting key 055bd34345d3d4178706f9d8bef95e81
WARNING: no policy specified for kudu/127.3.75.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab.
Entry for principal kudu/127.3.75.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab.
I20260926 01:03:24.022174 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.3:0
--local_ip_for_outbound_sockets=127.3.75.3
--webserver_interface=127.3.75.3
--webserver_port=0
--tserver_master_addrs=127.3.75.62:38481
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab
--principal=kudu/127.3.75.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:41297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:03:24.101328 20722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:03:24.101570 20722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:03:24.101609 20722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:03:24.102860 20722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:03:24.102941 20722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:03:24.102986 20722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.3
I20260926 01:03:24.104450 20722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:41297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab
--principal=kudu/127.3.75.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.75.3
--webserver_port=0
--tserver_master_addrs=127.3.75.62:38481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.3
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:03:24.104713 20722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:03:24.104938 20722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:03:24.110040 20731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:24.110026 20729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:24.110018 20727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:03:24.110287 20722 server_base.cc:1034] running on GCE node
I20260926 01:03:24.110505 20722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:03:24.110739 20722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:03:24.111907 20722 hybrid_clock.cc:648] HybridClock initialized: now 1790384604111889 us; error 53 us; skew 500 ppm
Sep 26 01:03:24 dist-test-slave-7v10 krb5kdc[20344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384604, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:03:24.114348 20722 init.cc:377] Logged in from keytab as kudu/127.3.75.3@KRBTEST.COM (short username kudu)
I20260926 01:03:24.114773 20722 webserver.cc:460] Webserver started at http://127.3.75.3:43421/ using document root <none> and password file <none>
I20260926 01:03:24.115031 20722 fs_manager.cc:359] Metadata directory not provided
I20260926 01:03:24.115103 20722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:03:24.115249 20722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:03:24.116343 20722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/ts-2/data/instance:
uuid: "2bd3550e557b47858d54f79d75a85825"
format_stamp: "Formatted at 2026-09-26 01:03:24 on dist-test-slave-7v10"
server_key: "d410ede33fff71ae5a755d1472a29b23"
server_key_iv: "59e0fe4a35c6c1b9de391ed140c9c28b"
server_key_version: "encryptionkey@0"
I20260926 01:03:24.116788 20722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/ts-2/wal/instance:
uuid: "2bd3550e557b47858d54f79d75a85825"
format_stamp: "Formatted at 2026-09-26 01:03:24 on dist-test-slave-7v10"
server_key: "d410ede33fff71ae5a755d1472a29b23"
server_key_iv: "59e0fe4a35c6c1b9de391ed140c9c28b"
server_key_version: "encryptionkey@0"
I20260926 01:03:24.118100 20722 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:03:24.118793 20738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:24.118990 20722 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:03:24.119091 20722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/ts-2/wal
uuid: "2bd3550e557b47858d54f79d75a85825"
format_stamp: "Formatted at 2026-09-26 01:03:24 on dist-test-slave-7v10"
server_key: "d410ede33fff71ae5a755d1472a29b23"
server_key_iv: "59e0fe4a35c6c1b9de391ed140c9c28b"
server_key_version: "encryptionkey@0"
I20260926 01:03:24.119179 20722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:03:24.126102 20722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:03:24.128209 20722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:03:24.128367 20722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:03:24.128594 20722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:03:24.128901 20722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:03:24.128955 20722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:24.128997 20722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:03:24.129025 20722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:24.135396 20722 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.3:45457
I20260926 01:03:24.135449 20852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.3:45457 every 8 connection(s)
I20260926 01:03:24.135751 20722 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
I20260926 01:03:24.136664 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 20722
I20260926 01:03:24.136806 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/ts-2/wal/instance
I20260926 01:03:24.137063 3372 external_mini_cluster.cc:1428] Setting key fe3ac7c915d55b84705f773e5888b109
Sep 26 01:03:24 dist-test-slave-7v10 krb5kdc[20344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384604, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.3@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 01:03:24.145340 20443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:24.137287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.3:41053 (local address 127.3.75.62:38481)
0926 01:03:24.137365 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:24.137368 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:24.137381 (+ 13us) server_negotiation.cc:407] Connection header received
0926 01:03:24.137591 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:24.137592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:24.137607 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:24.137680 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:24.138781 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:24.139408 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:24.140389 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:24.140625 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:24.140694 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:24.142756 (+ 2062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:24.142757 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:24.142758 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:03:24.142764 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:24.144094 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:24.144478 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:24.144479 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:24.144479 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:24.144559 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:24.144775 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:24.144775 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:24.144775 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:24.144875 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:24.144934 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:24.145191 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 01:03:24.145269 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 01:03:24.145443 20855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:24.136490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:38481 (local address 127.3.75.3:41053)
0926 01:03:24.136704 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:24.136759 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:24.137404 (+ 645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:24.137764 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:24.137774 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:24.138492 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:03:24.138688 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:24.138694 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:24.139502 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:24.139502 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:24.140298 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:24.140300 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:24.140709 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:24.140710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:24.140805 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:24.141265 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:03:24.141273 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:03:24.142648 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:03:24.144170 (+ 1522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:03:24.144176 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:03:24.144178 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:03:24.144413 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:03:24.144618 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:03:24.144618 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:03:24.144619 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:03:24.144716 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:03:24.144929 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:03:24.144933 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:03:24.145040 (+ 107us) client_negotiation.cc:769] Sending connection context
0926 01:03:24.145120 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 01:03:24.145248 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":119,"threads_started":1}
I20260926 01:03:24.146030 20853 heartbeater.cc:347] Connected to a master server at 127.3.75.62:38481
I20260926 01:03:24.146158 20853 heartbeater.cc:464] Registering TS with master...
I20260926 01:03:24.146377 20853 heartbeater.cc:511] Master 127.3.75.62:38481 requested a full tablet report, sending...
I20260926 01:03:24.146813 20389 ts_manager.cc:194] Registered new tserver with Master: 2bd3550e557b47858d54f79d75a85825 (127.3.75.3:45457)
I20260926 01:03:24.147236 20389 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.75.3@KRBTEST.COM'} at 127.3.75.3:41053
I20260926 01:03:24.149029 3372 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 01:03:24.156688 20443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:24.150628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52968 (local address 127.3.75.62:38481)
0926 01:03:24.150708 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:24.150711 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:24.150943 (+ 232us) server_negotiation.cc:407] Connection header received
0926 01:03:24.150977 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:24.150977 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:24.150987 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:24.151037 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:24.151595 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:24.152089 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:24.152810 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:24.152994 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:24.153056 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:24.154204 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:24.154205 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:24.154206 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:03:24.154211 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:24.155706 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:24.156004 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:24.156004 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:24.156005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:24.156055 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:24.156213 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:24.156214 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:24.156214 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:24.156346 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:24.156417 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:24.156518 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 01:03:24.156573 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 01:03:24.158421 20389 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52968:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:03:24.160105 20389 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:03:24.166329 20862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:24.162458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.1:42727 (local address 127.0.0.1:41808)
0926 01:03:24.162597 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:24.162629 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:24.162702 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:24.163097 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:24.163100 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:24.163102 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:03:24.163192 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:24.163194 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:24.164085 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:24.164086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:24.165274 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:24.165277 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:24.166043 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:24.166044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:24.166117 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:24.166130 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 01:03:24.166208 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 01:03:24.166258 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":4,"thread_start_us":83,"threads_started":1}
I20260926 01:03:24.166342 20866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:24.162587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41808 (local address 127.3.75.1:42727)
0926 01:03:24.162796 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:24.162800 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:24.162810 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:03:24.162860 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:24.162861 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:24.162927 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:24.163006 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:03:24.163282 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:24.163979 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:24.165369 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:24.165956 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:24.166041 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:24.166103 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:24.166224 (+ 121us) server_negotiation.cc:299] Negotiation successful
0926 01:03:24.166263 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":94,"threads_started":1}
I20260926 01:03:24.166713 20865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:24.162253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.2:39189 (local address 127.0.0.1:50892)
0926 01:03:24.162584 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:24.162627 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:24.162702 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:24.163100 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:24.163102 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:24.163104 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:03:24.163205 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:24.163207 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:24.164534 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:24.164535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:24.165574 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:24.165576 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:24.166520 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:24.166521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:24.166578 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:24.166582 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:03:24.166620 (+ 38us) client_negotiation.cc:240] Negotiation successful
0926 01:03:24.166646 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":30,"thread_start_us":75,"threads_started":1}
I20260926 01:03:24.166739 20863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:24.162357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41086 (local address 127.3.75.3:45457)
0926 01:03:24.162995 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:24.162999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:24.163008 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:03:24.163045 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:24.163047 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:24.163062 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:24.163133 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:03:24.163352 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:24.164442 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:24.165580 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:24.166286 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:24.166356 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:24.166409 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:24.166623 (+ 214us) server_negotiation.cc:299] Negotiation successful
0926 01:03:24.166664 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":97,"threads_started":1}
I20260926 01:03:24.167167 20864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:24.162401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50892 (local address 127.3.75.2:39189)
0926 01:03:24.162841 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:24.162851 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:24.162859 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:03:24.162932 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:24.162935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:24.162952 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:24.163022 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:03:24.163267 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:24.164416 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:24.165647 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:24.166298 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:24.166356 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:24.166977 (+ 621us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:24.167003 (+ 26us) server_negotiation.cc:299] Negotiation successful
0926 01:03:24.167056 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":83,"threads_started":1}
I20260926 01:03:24.166713 20861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:24.162253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.3:45457 (local address 127.0.0.1:41086)
0926 01:03:24.162584 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:24.162627 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:24.162696 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:24.163215 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:24.163216 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:24.163218 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:03:24.163263 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:24.163264 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:24.164546 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:24.164547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:24.165496 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:24.165499 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:24.166448 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:24.166449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:24.166525 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:24.166529 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:03:24.166584 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 01:03:24.166631 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":34,"thread_start_us":74,"threads_started":1}
I20260926 01:03:24.167424 20515 tablet_service.cc:1467] Processing CreateTablet for tablet 0afb0dc5bb7c458f9b24bf5435313353 (DEFAULT_TABLE table=test-table [id=f8d7c2d91de5484284a88a0d14e3203c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:03:24.167796 20515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0afb0dc5bb7c458f9b24bf5435313353. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:03:24.167969 20650 tablet_service.cc:1467] Processing CreateTablet for tablet 0afb0dc5bb7c458f9b24bf5435313353 (DEFAULT_TABLE table=test-table [id=f8d7c2d91de5484284a88a0d14e3203c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:03:24.168290 20650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0afb0dc5bb7c458f9b24bf5435313353. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:03:24.171482 20786 tablet_service.cc:1467] Processing CreateTablet for tablet 0afb0dc5bb7c458f9b24bf5435313353 (DEFAULT_TABLE table=test-table [id=f8d7c2d91de5484284a88a0d14e3203c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:03:24.171586 20867 tablet_bootstrap.cc:492] T 0afb0dc5bb7c458f9b24bf5435313353 P 39e8d4a39a604a68b0f2a8aee0bd848b: Bootstrap starting.
I20260926 01:03:24.171832 20786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0afb0dc5bb7c458f9b24bf5435313353. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:03:24.171913 20868 tablet_bootstrap.cc:492] T 0afb0dc5bb7c458f9b24bf5435313353 P c4d2a51a02b7433ebcff87c573ccc733: Bootstrap starting.
I20260926 01:03:24.172353 20867 tablet_bootstrap.cc:654] T 0afb0dc5bb7c458f9b24bf5435313353 P 39e8d4a39a604a68b0f2a8aee0bd848b: Neither blocks nor log segments found. Creating new log.
I20260926 01:03:24.172495 20868 tablet_bootstrap.cc:654] T 0afb0dc5bb7c458f9b24bf5435313353 P c4d2a51a02b7433ebcff87c573ccc733: Neither blocks nor log segments found. Creating new log.
I20260926 01:03:24.172962 20868 log.cc:824] T 0afb0dc5bb7c458f9b24bf5435313353 P c4d2a51a02b7433ebcff87c573ccc733: Log is configured to *not* fsync() on all Append() calls
I20260926 01:03:24.172955 20867 log.cc:824] T 0afb0dc5bb7c458f9b24bf5435313353 P 39e8d4a39a604a68b0f2a8aee0bd848b: Log is configured to *not* fsync() on all Append() calls
I20260926 01:03:24.173725 20868 tablet_bootstrap.cc:492] T 0afb0dc5bb7c458f9b24bf5435313353 P c4d2a51a02b7433ebcff87c573ccc733: No bootstrap required, opened a new log
I20260926 01:03:24.173820 20868 ts_tablet_manager.cc:1397] T 0afb0dc5bb7c458f9b24bf5435313353 P c4d2a51a02b7433ebcff87c573ccc733: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:03:24.173846 20867 tablet_bootstrap.cc:492] T 0afb0dc5bb7c458f9b24bf5435313353 P 39e8d4a39a604a68b0f2a8aee0bd848b: No bootstrap required, opened a new log
I20260926 01:03:24.173906 20867 ts_tablet_manager.cc:1397] T 0afb0dc5bb7c458f9b24bf5435313353 P 39e8d4a39a604a68b0f2a8aee0bd848b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 01:03:24.175233 20871 tablet_bootstrap.cc:492] T 0afb0dc5bb7c458f9b24bf5435313353 P 2bd3550e557b47858d54f79d75a85825: Bootstrap starting.
I20260926 01:03:24.175433 20868 raft_consensus.cc:348] T 0afb0dc5bb7c458f9b24bf5435313353 P c4d2a51a02b7433ebcff87c573ccc733 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d2a51a02b7433ebcff87c573ccc733" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39189 } } peers { permanent_uuid: "2bd3550e557b47858d54f79d75a85825" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 45457 } } peers { permanent_uuid: "39e8d4a39a604a68b0f2a8aee0bd848b" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 42727 } }
I20260926 01:03:24.175634 20868 raft_consensus.cc:374] T 0afb0dc5bb7c458f9b24bf5435313353 P c4d2a51a02b7433ebcff87c573ccc733 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:03:24.175669 20868 raft_consensus.cc:729] T 0afb0dc5bb7c458f9b24bf5435313353 P c4d2a51a02b7433ebcff87c573ccc733 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4d2a51a02b7433ebcff87c573ccc733, State: Initialized, Role: FOLLOWER
I20260926 01:03:24.175794 20868 consensus_queue.cc:260] T 0afb0dc5bb7c458f9b24bf5435313353 P c4d2a51a02b7433ebcff87c573ccc733 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d2a51a02b7433ebcff87c573ccc733" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39189 } } peers { permanent_uuid: "2bd3550e557b47858d54f79d75a85825" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 45457 } } peers { permanent_uuid: "39e8d4a39a604a68b0f2a8aee0bd848b" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 42727 } }
I20260926 01:03:24.175899 20871 tablet_bootstrap.cc:654] T 0afb0dc5bb7c458f9b24bf5435313353 P 2bd3550e557b47858d54f79d75a85825: Neither blocks nor log segments found. Creating new log.
I20260926 01:03:24.176105 20868 ts_tablet_manager.cc:1428] T 0afb0dc5bb7c458f9b24bf5435313353 P c4d2a51a02b7433ebcff87c573ccc733: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 01:03:24.176038 20867 raft_consensus.cc:348] T 0afb0dc5bb7c458f9b24bf5435313353 P 39e8d4a39a604a68b0f2a8aee0bd848b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d2a51a02b7433ebcff87c573ccc733" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39189 } } peers { permanent_uuid: "2bd3550e557b47858d54f79d75a85825" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 45457 } } peers { permanent_uuid: "39e8d4a39a604a68b0f2a8aee0bd848b" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 42727 } }
I20260926 01:03:24.176203 20867 raft_consensus.cc:374] T 0afb0dc5bb7c458f9b24bf5435313353 P 39e8d4a39a604a68b0f2a8aee0bd848b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:03:24.176250 20867 raft_consensus.cc:729] T 0afb0dc5bb7c458f9b24bf5435313353 P 39e8d4a39a604a68b0f2a8aee0bd848b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39e8d4a39a604a68b0f2a8aee0bd848b, State: Initialized, Role: FOLLOWER
I20260926 01:03:24.176285 20717 heartbeater.cc:503] Master 127.3.75.62:38481 was elected leader, sending a full tablet report...
I20260926 01:03:24.176389 20867 consensus_queue.cc:260] T 0afb0dc5bb7c458f9b24bf5435313353 P 39e8d4a39a604a68b0f2a8aee0bd848b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d2a51a02b7433ebcff87c573ccc733" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39189 } } peers { permanent_uuid: "2bd3550e557b47858d54f79d75a85825" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 45457 } } peers { permanent_uuid: "39e8d4a39a604a68b0f2a8aee0bd848b" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 42727 } }
I20260926 01:03:24.176478 20871 log.cc:824] T 0afb0dc5bb7c458f9b24bf5435313353 P 2bd3550e557b47858d54f79d75a85825: Log is configured to *not* fsync() on all Append() calls
I20260926 01:03:24.176654 20867 ts_tablet_manager.cc:1428] T 0afb0dc5bb7c458f9b24bf5435313353 P 39e8d4a39a604a68b0f2a8aee0bd848b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 01:03:24.176789 20581 heartbeater.cc:503] Master 127.3.75.62:38481 was elected leader, sending a full tablet report...
I20260926 01:03:24.177350 20871 tablet_bootstrap.cc:492] T 0afb0dc5bb7c458f9b24bf5435313353 P 2bd3550e557b47858d54f79d75a85825: No bootstrap required, opened a new log
I20260926 01:03:24.177454 20871 ts_tablet_manager.cc:1397] T 0afb0dc5bb7c458f9b24bf5435313353 P 2bd3550e557b47858d54f79d75a85825: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 01:03:24.179036 20871 raft_consensus.cc:348] T 0afb0dc5bb7c458f9b24bf5435313353 P 2bd3550e557b47858d54f79d75a85825 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d2a51a02b7433ebcff87c573ccc733" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39189 } } peers { permanent_uuid: "2bd3550e557b47858d54f79d75a85825" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 45457 } } peers { permanent_uuid: "39e8d4a39a604a68b0f2a8aee0bd848b" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 42727 } }
I20260926 01:03:24.179158 20871 raft_consensus.cc:374] T 0afb0dc5bb7c458f9b24bf5435313353 P 2bd3550e557b47858d54f79d75a85825 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:03:24.179186 20871 raft_consensus.cc:729] T 0afb0dc5bb7c458f9b24bf5435313353 P 2bd3550e557b47858d54f79d75a85825 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bd3550e557b47858d54f79d75a85825, State: Initialized, Role: FOLLOWER
I20260926 01:03:24.179327 20871 consensus_queue.cc:260] T 0afb0dc5bb7c458f9b24bf5435313353 P 2bd3550e557b47858d54f79d75a85825 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d2a51a02b7433ebcff87c573ccc733" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39189 } } peers { permanent_uuid: "2bd3550e557b47858d54f79d75a85825" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 45457 } } peers { permanent_uuid: "39e8d4a39a604a68b0f2a8aee0bd848b" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 42727 } }
I20260926 01:03:24.179606 20871 ts_tablet_manager.cc:1428] T 0afb0dc5bb7c458f9b24bf5435313353 P 2bd3550e557b47858d54f79d75a85825: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 01:03:24.179628 20853 heartbeater.cc:503] Master 127.3.75.62:38481 was elected leader, sending a full tablet report...
W20260926 01:03:24.239094 20718 tablet.cc:2369] T 0afb0dc5bb7c458f9b24bf5435313353 P c4d2a51a02b7433ebcff87c573ccc733: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 01:03:24.335953 20582 tablet.cc:2369] T 0afb0dc5bb7c458f9b24bf5435313353 P 39e8d4a39a604a68b0f2a8aee0bd848b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 01:03:24.386296 20854 tablet.cc:2369] T 0afb0dc5bb7c458f9b24bf5435313353 P 2bd3550e557b47858d54f79d75a85825: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:03:24.406148 20873 raft_consensus.cc:482] T 0afb0dc5bb7c458f9b24bf5435313353 P 39e8d4a39a604a68b0f2a8aee0bd848b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:03:24.406288 20873 raft_consensus.cc:504] T 0afb0dc5bb7c458f9b24bf5435313353 P 39e8d4a39a604a68b0f2a8aee0bd848b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d2a51a02b7433ebcff87c573ccc733" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39189 } } peers { permanent_uuid: "2bd3550e557b47858d54f79d75a85825" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 45457 } } peers { permanent_uuid: "39e8d4a39a604a68b0f2a8aee0bd848b" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 42727 } }
I20260926 01:03:24.406704 20873 leader_election.cc:288] T 0afb0dc5bb7c458f9b24bf5435313353 P 39e8d4a39a604a68b0f2a8aee0bd848b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4d2a51a02b7433ebcff87c573ccc733 (127.3.75.2:39189), 2bd3550e557b47858d54f79d75a85825 (127.3.75.3:45457)
I20260926 01:03:24.409838 20864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:24.407045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:43169 (local address 127.3.75.2:39189)
0926 01:03:24.407095 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:24.407097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:24.407175 (+ 78us) server_negotiation.cc:407] Connection header received
0926 01:03:24.407237 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:24.407238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:24.407250 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:24.407337 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:03:24.407588 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:24.408044 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:24.409105 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:24.409472 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:24.409526 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:24.409534 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:24.409711 (+ 177us) server_negotiation.cc:299] Negotiation successful
0926 01:03:24.409757 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 01:03:24.409869 20863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:24.407045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:39585 (local address 127.3.75.3:45457)
0926 01:03:24.407095 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:24.407097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:24.407175 (+ 78us) server_negotiation.cc:407] Connection header received
0926 01:03:24.407237 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:24.407238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:24.407250 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:24.407328 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:03:24.407533 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:24.408026 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:24.409118 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:24.409490 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:24.409574 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:24.409584 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:24.409752 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 01:03:24.409796 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 01:03:24.409904 20876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:24.406953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.3:45457 (local address 127.3.75.1:39585)
0926 01:03:24.407091 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:24.407103 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:24.407183 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:24.407355 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:24.407356 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:24.407358 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:03:24.407442 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:24.407444 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:24.408112 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:24.408114 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:24.409012 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:24.409015 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:24.409603 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:24.409604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:24.409658 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:24.409661 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:03:24.409734 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 01:03:24.409773 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":54,"thread_start_us":70,"threads_started":1}
I20260926 01:03:24.410094 20877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:24.406953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.2:39189 (local address 127.3.75.1:43169)
0926 01:03:24.407091 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:24.407103 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:24.407173 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:24.407388 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:24.407390 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:24.407392 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:03:24.407484 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:24.407504 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:24.408148 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:24.408153 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:24.409010 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:24.409012 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:24.409552 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:24.409553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:24.409629 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:24.409633 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:03:24.409692 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 01:03:24.409750 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":25,"thread_start_us":69,"threads_started":1}
I20260926 01:03:24.410265 20807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afb0dc5bb7c458f9b24bf5435313353" candidate_uuid: "39e8d4a39a604a68b0f2a8aee0bd848b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bd3550e557b47858d54f79d75a85825" is_pre_election: true
I20260926 01:03:24.410427 20807 raft_consensus.cc:2454] T 0afb0dc5bb7c458f9b24bf5435313353 P 2bd3550e557b47858d54f79d75a85825 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39e8d4a39a604a68b0f2a8aee0bd848b in term 0.
I20260926 01:03:24.410595 20671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afb0dc5bb7c458f9b24bf5435313353" candidate_uuid: "39e8d4a39a604a68b0f2a8aee0bd848b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4d2a51a02b7433ebcff87c573ccc733" is_pre_election: true
I20260926 01:03:24.410638 20469 leader_election.cc:302] T 0afb0dc5bb7c458f9b24bf5435313353 P 39e8d4a39a604a68b0f2a8aee0bd848b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bd3550e557b47858d54f79d75a85825, 39e8d4a39a604a68b0f2a8aee0bd848b; no voters:
I20260926 01:03:24.410749 20671 raft_consensus.cc:2454] T 0afb0dc5bb7c458f9b24bf5435313353 P c4d2a51a02b7433ebcff87c573ccc733 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39e8d4a39a604a68b0f2a8aee0bd848b in term 0.
I20260926 01:03:24.410826 20873 raft_consensus.cc:2781] T 0afb0dc5bb7c458f9b24bf5435313353 P 39e8d4a39a604a68b0f2a8aee0bd848b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:03:24.410928 20873 raft_consensus.cc:482] T 0afb0dc5bb7c458f9b24bf5435313353 P 39e8d4a39a604a68b0f2a8aee0bd848b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:03:24.410971 20873 raft_consensus.cc:3037] T 0afb0dc5bb7c458f9b24bf5435313353 P 39e8d4a39a604a68b0f2a8aee0bd848b [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:03:24.411672 20873 raft_consensus.cc:504] T 0afb0dc5bb7c458f9b24bf5435313353 P 39e8d4a39a604a68b0f2a8aee0bd848b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d2a51a02b7433ebcff87c573ccc733" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39189 } } peers { permanent_uuid: "2bd3550e557b47858d54f79d75a85825" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 45457 } } peers { permanent_uuid: "39e8d4a39a604a68b0f2a8aee0bd848b" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 42727 } }
I20260926 01:03:24.411814 20873 leader_election.cc:288] T 0afb0dc5bb7c458f9b24bf5435313353 P 39e8d4a39a604a68b0f2a8aee0bd848b [CANDIDATE]: Term 1 election: Requested vote from peers c4d2a51a02b7433ebcff87c573ccc733 (127.3.75.2:39189), 2bd3550e557b47858d54f79d75a85825 (127.3.75.3:45457)
I20260926 01:03:24.411953 20671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afb0dc5bb7c458f9b24bf5435313353" candidate_uuid: "39e8d4a39a604a68b0f2a8aee0bd848b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4d2a51a02b7433ebcff87c573ccc733"
I20260926 01:03:24.411990 20807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afb0dc5bb7c458f9b24bf5435313353" candidate_uuid: "39e8d4a39a604a68b0f2a8aee0bd848b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bd3550e557b47858d54f79d75a85825"
I20260926 01:03:24.412034 20671 raft_consensus.cc:3037] T 0afb0dc5bb7c458f9b24bf5435313353 P c4d2a51a02b7433ebcff87c573ccc733 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:03:24.412055 20807 raft_consensus.cc:3037] T 0afb0dc5bb7c458f9b24bf5435313353 P 2bd3550e557b47858d54f79d75a85825 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:03:24.412966 20671 raft_consensus.cc:2454] T 0afb0dc5bb7c458f9b24bf5435313353 P c4d2a51a02b7433ebcff87c573ccc733 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39e8d4a39a604a68b0f2a8aee0bd848b in term 1.
I20260926 01:03:24.412966 20807 raft_consensus.cc:2454] T 0afb0dc5bb7c458f9b24bf5435313353 P 2bd3550e557b47858d54f79d75a85825 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39e8d4a39a604a68b0f2a8aee0bd848b in term 1.
I20260926 01:03:24.413172 20468 leader_election.cc:302] T 0afb0dc5bb7c458f9b24bf5435313353 P 39e8d4a39a604a68b0f2a8aee0bd848b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39e8d4a39a604a68b0f2a8aee0bd848b, c4d2a51a02b7433ebcff87c573ccc733; no voters:
I20260926 01:03:24.413303 20873 raft_consensus.cc:2781] T 0afb0dc5bb7c458f9b24bf5435313353 P 39e8d4a39a604a68b0f2a8aee0bd848b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:03:24.413528 20873 raft_consensus.cc:686] T 0afb0dc5bb7c458f9b24bf5435313353 P 39e8d4a39a604a68b0f2a8aee0bd848b [term 1 LEADER]: Becoming Leader. State: Replica: 39e8d4a39a604a68b0f2a8aee0bd848b, State: Running, Role: LEADER
I20260926 01:03:24.413641 20873 consensus_queue.cc:237] T 0afb0dc5bb7c458f9b24bf5435313353 P 39e8d4a39a604a68b0f2a8aee0bd848b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d2a51a02b7433ebcff87c573ccc733" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39189 } } peers { permanent_uuid: "2bd3550e557b47858d54f79d75a85825" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 45457 } } peers { permanent_uuid: "39e8d4a39a604a68b0f2a8aee0bd848b" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 42727 } }
I20260926 01:03:24.414409 20389 catalog_manager.cc:5495] T 0afb0dc5bb7c458f9b24bf5435313353 P 39e8d4a39a604a68b0f2a8aee0bd848b reported cstate change: term changed from 0 to 1, leader changed from <none> to 39e8d4a39a604a68b0f2a8aee0bd848b (127.3.75.1). New cstate: current_term: 1 leader_uuid: "39e8d4a39a604a68b0f2a8aee0bd848b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d2a51a02b7433ebcff87c573ccc733" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2bd3550e557b47858d54f79d75a85825" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 45457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39e8d4a39a604a68b0f2a8aee0bd848b" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 42727 } health_report { overall_health: HEALTHY } } }
I20260926 01:03:24.447750 20866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:24.445585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41822 (local address 127.3.75.1:42727)
0926 01:03:24.445655 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:24.445657 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:24.445664 (+ 7us) server_negotiation.cc:407] Connection header received
0926 01:03:24.445700 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:24.445701 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:24.445719 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:24.445779 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:03:24.445894 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:24.446361 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:24.447101 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:24.447293 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:24.447342 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:24.447476 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:03:24.447570 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:03:24.447628 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:24.447678 (+ 50us) server_negotiation.cc:299] Negotiation successful
0926 01:03:24.447696 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 01:03:24.449579 20671 raft_consensus.cc:1261] T 0afb0dc5bb7c458f9b24bf5435313353 P c4d2a51a02b7433ebcff87c573ccc733 [term 1 FOLLOWER]: Refusing update from remote peer 39e8d4a39a604a68b0f2a8aee0bd848b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:03:24.449581 20807 raft_consensus.cc:1261] T 0afb0dc5bb7c458f9b24bf5435313353 P 2bd3550e557b47858d54f79d75a85825 [term 1 FOLLOWER]: Refusing update from remote peer 39e8d4a39a604a68b0f2a8aee0bd848b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:03:24.449960 20878 consensus_queue.cc:1035] T 0afb0dc5bb7c458f9b24bf5435313353 P 39e8d4a39a604a68b0f2a8aee0bd848b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bd3550e557b47858d54f79d75a85825" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 45457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:03:24.450115 20873 consensus_queue.cc:1035] T 0afb0dc5bb7c458f9b24bf5435313353 P 39e8d4a39a604a68b0f2a8aee0bd848b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d2a51a02b7433ebcff87c573ccc733" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 39189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:03:24.453670 20883 mvcc.cc:205] Tried to move back new op lower bound from 7333415339822350336 to 7333415339678904320. Current Snapshot: MvccSnapshot[applied={T|T < 7333415339822350336}]
I20260926 01:03:24.454937 20388 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:03:24.455049 20388 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:03:24.455835 20388 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8088e19ce2b940f1bf94d8200c557963: Sending DeleteTablet for 3 replicas of tablet 0afb0dc5bb7c458f9b24bf5435313353
I20260926 01:03:24.456156 20786 tablet_service.cc:1514] Processing DeleteTablet for tablet 0afb0dc5bb7c458f9b24bf5435313353 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:03:24 UTC) from {username='kudu', principal='kudu/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:41086
I20260926 01:03:24.456171 20650 tablet_service.cc:1514] Processing DeleteTablet for tablet 0afb0dc5bb7c458f9b24bf5435313353 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:03:24 UTC) from {username='kudu', principal='kudu/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:50892
I20260926 01:03:24.456184 20515 tablet_service.cc:1514] Processing DeleteTablet for tablet 0afb0dc5bb7c458f9b24bf5435313353 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:03:24 UTC) from {username='kudu', principal='kudu/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:41808
I20260926 01:03:24.456395 20892 tablet_replica.cc:330] T 0afb0dc5bb7c458f9b24bf5435313353 P 2bd3550e557b47858d54f79d75a85825: stopping tablet replica
I20260926 01:03:24.456444 20891 tablet_replica.cc:330] T 0afb0dc5bb7c458f9b24bf5435313353 P c4d2a51a02b7433ebcff87c573ccc733: stopping tablet replica
I20260926 01:03:24.456530 20892 raft_consensus.cc:2229] T 0afb0dc5bb7c458f9b24bf5435313353 P 2bd3550e557b47858d54f79d75a85825 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:03:24.456540 20891 raft_consensus.cc:2229] T 0afb0dc5bb7c458f9b24bf5435313353 P c4d2a51a02b7433ebcff87c573ccc733 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:03:24.456646 20892 raft_consensus.cc:2258] T 0afb0dc5bb7c458f9b24bf5435313353 P 2bd3550e557b47858d54f79d75a85825 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:03:24.456790 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 20452
I20260926 01:03:24.457125 20892 ts_tablet_manager.cc:1905] T 0afb0dc5bb7c458f9b24bf5435313353 P 2bd3550e557b47858d54f79d75a85825: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:03:24.459110 20892 ts_tablet_manager.cc:1918] T 0afb0dc5bb7c458f9b24bf5435313353 P 2bd3550e557b47858d54f79d75a85825: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:03:24.459189 20892 log.cc:1196] T 0afb0dc5bb7c458f9b24bf5435313353 P 2bd3550e557b47858d54f79d75a85825: Deleting WAL directory at /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/ts-2/wal/wals/0afb0dc5bb7c458f9b24bf5435313353
I20260926 01:03:24.459491 20892 ts_tablet_manager.cc:1939] T 0afb0dc5bb7c458f9b24bf5435313353 P 2bd3550e557b47858d54f79d75a85825: Deleting consensus metadata
I20260926 01:03:24.459892 20374 catalog_manager.cc:4841] TS 2bd3550e557b47858d54f79d75a85825 (127.3.75.3:45457): tablet 0afb0dc5bb7c458f9b24bf5435313353 (table test-table [id=f8d7c2d91de5484284a88a0d14e3203c]) successfully deleted
I20260926 01:03:24.461697 20891 raft_consensus.cc:2258] T 0afb0dc5bb7c458f9b24bf5435313353 P c4d2a51a02b7433ebcff87c573ccc733 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:03:24.462074 20891 ts_tablet_manager.cc:1905] T 0afb0dc5bb7c458f9b24bf5435313353 P c4d2a51a02b7433ebcff87c573ccc733: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:03:24.463320 20891 ts_tablet_manager.cc:1918] T 0afb0dc5bb7c458f9b24bf5435313353 P c4d2a51a02b7433ebcff87c573ccc733: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:03:24.463409 20891 log.cc:1196] T 0afb0dc5bb7c458f9b24bf5435313353 P c4d2a51a02b7433ebcff87c573ccc733: Deleting WAL directory at /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384372073861-3372-0/minicluster-data/ts-1/wal/wals/0afb0dc5bb7c458f9b24bf5435313353
I20260926 01:03:24.463732 20891 ts_tablet_manager.cc:1939] T 0afb0dc5bb7c458f9b24bf5435313353 P c4d2a51a02b7433ebcff87c573ccc733: Deleting consensus metadata
I20260926 01:03:24.464093 20373 catalog_manager.cc:4841] TS c4d2a51a02b7433ebcff87c573ccc733 (127.3.75.2:39189): tablet 0afb0dc5bb7c458f9b24bf5435313353 (table test-table [id=f8d7c2d91de5484284a88a0d14e3203c]) successfully deleted
W20260926 01:03:24.465194 20374 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.75.1:42727 (error 108)
I20260926 01:03:24.465386 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 20586
I20260926 01:03:24.466032 20865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:24.465582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.1:42727 (local address 127.0.0.1:41828)
0926 01:03:24.465960 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:24.466002 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.1:42727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356}
W20260926 01:03:24.466770 20374 catalog_manager.cc:4584] TS 39e8d4a39a604a68b0f2a8aee0bd848b (127.3.75.1:42727): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0afb0dc5bb7c458f9b24bf5435313353: Network error: Client connection negotiation failed: client connection to 127.3.75.1:42727: connect: Connection refused (error 111)
I20260926 01:03:24.470598 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 20722
I20260926 01:03:24.475728 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 20357
2026-09-26T01:03:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2100 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-26T01:03:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:03:24Z Disabled control of system clock
I20260926 01:03:24.503041 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:33633
--webserver_interface=127.3.75.62
--webserver_port=0
--builtin_ntp_servers=127.3.75.20:46647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.75.62:33633
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 01:03:24.588583 20897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:03:24.588780 20897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:03:24.588800 20897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:03:24.589942 20897 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:03:24.589978 20897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:03:24.589990 20897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:03:24.590003 20897 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:03:24.590013 20897 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:03:24.591336 20897 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:46647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.75.62:33633
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:33633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:03:24.591521 20897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:03:24.591761 20897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:03:24.596491 20903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:24.596498 20905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:03:24.596679 20897 server_base.cc:1034] running on GCE node
W20260926 01:03:24.596567 20902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:03:24.597021 20897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:03:24.597319 20897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:03:24.598515 20897 hybrid_clock.cc:648] HybridClock initialized: now 1790384604598491 us; error 33 us; skew 500 ppm
I20260926 01:03:24.599839 20897 webserver.cc:460] Webserver started at http://127.3.75.62:44011/ using document root <none> and password file <none>
I20260926 01:03:24.600114 20897 fs_manager.cc:359] Metadata directory not provided
I20260926 01:03:24.600164 20897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:03:24.600337 20897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:03:24.601372 20897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/master-0/data/instance:
uuid: "b723884a155a447e866a20a6cda7f211"
format_stamp: "Formatted at 2026-09-26 01:03:24 on dist-test-slave-7v10"
server_key: "a99957affedc15a5c090b0701d2bc005"
server_key_iv: "ed3325e0149b79e09c21d6c557583c41"
server_key_version: "encryptionkey@0"
I20260926 01:03:24.601715 20897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/master-0/wal/instance:
uuid: "b723884a155a447e866a20a6cda7f211"
format_stamp: "Formatted at 2026-09-26 01:03:24 on dist-test-slave-7v10"
server_key: "a99957affedc15a5c090b0701d2bc005"
server_key_iv: "ed3325e0149b79e09c21d6c557583c41"
server_key_version: "encryptionkey@0"
I20260926 01:03:24.603018 20897 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 01:03:24.603744 20911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:24.603925 20897 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:03:24.604010 20897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/master-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/master-0/wal
uuid: "b723884a155a447e866a20a6cda7f211"
format_stamp: "Formatted at 2026-09-26 01:03:24 on dist-test-slave-7v10"
server_key: "a99957affedc15a5c090b0701d2bc005"
server_key_iv: "ed3325e0149b79e09c21d6c557583c41"
server_key_version: "encryptionkey@0"
I20260926 01:03:24.604063 20897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:03:24.615584 20897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:03:24.615839 20897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:03:24.615936 20897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:03:24.619654 20897 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.62:33633
I20260926 01:03:24.619685 20964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.62:33633 every 8 connection(s)
I20260926 01:03:24.620038 20897 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
I20260926 01:03:24.620326 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 20897
I20260926 01:03:24.620433 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/master-0/wal/instance
I20260926 01:03:24.620641 3372 external_mini_cluster.cc:1428] Setting key 83b37d85d4f63f8feaba9a5a3701ea2f
I20260926 01:03:24.621233 20965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:03:24.624380 20965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b723884a155a447e866a20a6cda7f211: Bootstrap starting.
I20260926 01:03:24.624939 20965 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b723884a155a447e866a20a6cda7f211: Neither blocks nor log segments found. Creating new log.
I20260926 01:03:24.625231 20965 log.cc:824] T 00000000000000000000000000000000 P b723884a155a447e866a20a6cda7f211: Log is configured to *not* fsync() on all Append() calls
I20260926 01:03:24.625933 20965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b723884a155a447e866a20a6cda7f211: No bootstrap required, opened a new log
I20260926 01:03:24.626523 20968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:24.621170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41220 (local address 127.3.75.62:33633)
0926 01:03:24.621411 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:24.621420 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:24.621447 (+ 27us) server_negotiation.cc:407] Connection header received
0926 01:03:24.621969 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:24.621978 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:24.622058 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:24.622214 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:24.622420 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:24.623773 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:24.624551 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:24.624961 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:24.625100 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:24.625869 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:24.625882 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:24.625882 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:03:24.626071 (+ 189us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:24.626109 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:03:24.626129 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:24.626201 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:24.626278 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 01:03:24.626356 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":80,"threads_started":1}
I20260926 01:03:24.627967 20965 raft_consensus.cc:348] T 00000000000000000000000000000000 P b723884a155a447e866a20a6cda7f211 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b723884a155a447e866a20a6cda7f211" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 33633 } }
I20260926 01:03:24.628121 20965 raft_consensus.cc:374] T 00000000000000000000000000000000 P b723884a155a447e866a20a6cda7f211 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:03:24.628172 20965 raft_consensus.cc:729] T 00000000000000000000000000000000 P b723884a155a447e866a20a6cda7f211 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b723884a155a447e866a20a6cda7f211, State: Initialized, Role: FOLLOWER
I20260926 01:03:24.628297 20965 consensus_queue.cc:260] T 00000000000000000000000000000000 P b723884a155a447e866a20a6cda7f211 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b723884a155a447e866a20a6cda7f211" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 33633 } }
I20260926 01:03:24.628376 20965 raft_consensus.cc:388] T 00000000000000000000000000000000 P b723884a155a447e866a20a6cda7f211 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:03:24.628419 20965 raft_consensus.cc:482] T 00000000000000000000000000000000 P b723884a155a447e866a20a6cda7f211 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:03:24.628472 20965 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b723884a155a447e866a20a6cda7f211 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:03:24.628990 20965 raft_consensus.cc:504] T 00000000000000000000000000000000 P b723884a155a447e866a20a6cda7f211 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b723884a155a447e866a20a6cda7f211" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 33633 } }
I20260926 01:03:24.629104 20965 leader_election.cc:302] T 00000000000000000000000000000000 P b723884a155a447e866a20a6cda7f211 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b723884a155a447e866a20a6cda7f211; no voters:
I20260926 01:03:24.629289 20965 leader_election.cc:288] T 00000000000000000000000000000000 P b723884a155a447e866a20a6cda7f211 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:03:24.629323 20970 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b723884a155a447e866a20a6cda7f211 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:03:24.629479 20970 raft_consensus.cc:686] T 00000000000000000000000000000000 P b723884a155a447e866a20a6cda7f211 [term 1 LEADER]: Becoming Leader. State: Replica: b723884a155a447e866a20a6cda7f211, State: Running, Role: LEADER
I20260926 01:03:24.629555 20965 sys_catalog.cc:564] T 00000000000000000000000000000000 P b723884a155a447e866a20a6cda7f211 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:03:24.629559 20970 consensus_queue.cc:237] T 00000000000000000000000000000000 P b723884a155a447e866a20a6cda7f211 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b723884a155a447e866a20a6cda7f211" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 33633 } }
I20260926 01:03:24.629969 20972 sys_catalog.cc:455] T 00000000000000000000000000000000 P b723884a155a447e866a20a6cda7f211 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b723884a155a447e866a20a6cda7f211. Latest consensus state: current_term: 1 leader_uuid: "b723884a155a447e866a20a6cda7f211" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b723884a155a447e866a20a6cda7f211" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 33633 } } }
I20260926 01:03:24.630064 20972 sys_catalog.cc:458] T 00000000000000000000000000000000 P b723884a155a447e866a20a6cda7f211 [sys.catalog]: This master's current role is: LEADER
I20260926 01:03:24.629964 20971 sys_catalog.cc:455] T 00000000000000000000000000000000 P b723884a155a447e866a20a6cda7f211 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b723884a155a447e866a20a6cda7f211" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b723884a155a447e866a20a6cda7f211" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 33633 } } }
I20260926 01:03:24.630246 20971 sys_catalog.cc:458] T 00000000000000000000000000000000 P b723884a155a447e866a20a6cda7f211 [sys.catalog]: This master's current role is: LEADER
I20260926 01:03:24.630517 20977 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:03:24.631017 20977 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:03:24.632490 20977 catalog_manager.cc:1347] Generated new cluster ID: fa2c1da3b88b49a6afa60f7e58ef71f5
I20260926 01:03:24.632543 20977 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:03:24.642125 20977 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:03:24.642621 20977 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:03:24.646934 20977 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b723884a155a447e866a20a6cda7f211: Generated new TSK 0
I20260926 01:03:24.647123 20977 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:03:24.648319 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.1:0
--local_ip_for_outbound_sockets=127.3.75.1
--webserver_interface=127.3.75.1
--webserver_port=0
--tserver_master_addrs=127.3.75.62:33633
--builtin_ntp_servers=127.3.75.20:46647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 01:03:24.729772 20989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:03:24.730038 20989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:03:24.730073 20989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:03:24.731447 20989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:03:24.731504 20989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:03:24.731549 20989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.1
I20260926 01:03:24.733018 20989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:46647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.75.1
--webserver_port=0
--tserver_master_addrs=127.3.75.62:33633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.1
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:03:24.733273 20989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:03:24.733590 20989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:03:24.738621 20994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:03:24.738765 20989 server_base.cc:1034] running on GCE node
W20260926 01:03:24.738664 20995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:24.738647 20997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:03:24.739133 20989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:03:24.739434 20989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:03:24.740633 20989 hybrid_clock.cc:648] HybridClock initialized: now 1790384604740619 us; error 30 us; skew 500 ppm
I20260926 01:03:24.741918 20989 webserver.cc:460] Webserver started at http://127.3.75.1:42487/ using document root <none> and password file <none>
I20260926 01:03:24.742156 20989 fs_manager.cc:359] Metadata directory not provided
I20260926 01:03:24.742225 20989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:03:24.742374 20989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:03:24.743505 20989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/ts-0/data/instance:
uuid: "d469a42ae6d446349aadea90ccceba57"
format_stamp: "Formatted at 2026-09-26 01:03:24 on dist-test-slave-7v10"
server_key: "7c2bd5b0d59db7cb6cdc09500d16a5ad"
server_key_iv: "ef304d7eb638112f9ca6de44b53897a4"
server_key_version: "encryptionkey@0"
I20260926 01:03:24.743870 20989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/ts-0/wal/instance:
uuid: "d469a42ae6d446349aadea90ccceba57"
format_stamp: "Formatted at 2026-09-26 01:03:24 on dist-test-slave-7v10"
server_key: "7c2bd5b0d59db7cb6cdc09500d16a5ad"
server_key_iv: "ef304d7eb638112f9ca6de44b53897a4"
server_key_version: "encryptionkey@0"
I20260926 01:03:24.745185 20989 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:03:24.745914 21003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:24.746069 20989 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:03:24.746184 20989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/ts-0/wal
uuid: "d469a42ae6d446349aadea90ccceba57"
format_stamp: "Formatted at 2026-09-26 01:03:24 on dist-test-slave-7v10"
server_key: "7c2bd5b0d59db7cb6cdc09500d16a5ad"
server_key_iv: "ef304d7eb638112f9ca6de44b53897a4"
server_key_version: "encryptionkey@0"
I20260926 01:03:24.746255 20989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:03:24.755067 20989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:03:24.755336 20989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:03:24.755476 20989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:03:24.755704 20989 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:03:24.755998 20989 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:03:24.756031 20989 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:24.756050 20989 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:03:24.756095 20989 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:24.761874 20989 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.1:37353
I20260926 01:03:24.761921 21117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.1:37353 every 8 connection(s)
I20260926 01:03:24.762219 20989 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
I20260926 01:03:24.763343 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 20989
I20260926 01:03:24.763429 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/ts-0/wal/instance
I20260926 01:03:24.763641 3372 external_mini_cluster.cc:1428] Setting key 5601ff9affb79de146f6237a273c8f87
I20260926 01:03:24.764639 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.2:0
--local_ip_for_outbound_sockets=127.3.75.2
--webserver_interface=127.3.75.2
--webserver_port=0
--tserver_master_addrs=127.3.75.62:33633
--builtin_ntp_servers=127.3.75.20:46647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 01:03:24.768150 20968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:24.762921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:51191 (local address 127.3.75.62:33633)
0926 01:03:24.762983 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:24.762985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:24.763287 (+ 302us) server_negotiation.cc:407] Connection header received
0926 01:03:24.763965 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:24.763966 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:24.763981 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:24.764054 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:24.764868 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:24.765642 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:24.766603 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:24.766962 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:24.767030 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:24.767897 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:24.767898 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:24.767898 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:03:24.767903 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:24.767916 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:03:24.767920 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:24.767967 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:24.768043 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 01:03:24.768072 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 01:03:24.768371 21120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:24.762937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:33633 (local address 127.3.75.1:51191)
0926 01:03:24.763143 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:24.763201 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:24.763858 (+ 657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:24.764127 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:24.764131 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:24.764141 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:03:24.764789 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:24.764797 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:24.765731 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:24.765731 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:24.766519 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:24.766521 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:24.767044 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:24.767045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:24.767163 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:24.767732 (+ 569us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 01:03:24.767733 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:03:24.767768 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 01:03:24.767789 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:03:24.767966 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:03:24.767972 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:03:24.767975 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:03:24.768045 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 01:03:24.768141 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":99,"threads_started":1}
I20260926 01:03:24.768795 21118 heartbeater.cc:347] Connected to a master server at 127.3.75.62:33633
I20260926 01:03:24.768885 21118 heartbeater.cc:464] Registering TS with master...
I20260926 01:03:24.769057 21118 heartbeater.cc:511] Master 127.3.75.62:33633 requested a full tablet report, sending...
I20260926 01:03:24.769582 20928 ts_manager.cc:194] Registered new tserver with Master: d469a42ae6d446349aadea90ccceba57 (127.3.75.1:37353)
I20260926 01:03:24.770193 20928 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.75.1:51191
W20260926 01:03:24.846333 21121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:03:24.846532 21121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:03:24.846553 21121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:03:24.847656 21121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:03:24.847692 21121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:03:24.847716 21121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.2
I20260926 01:03:24.849092 21121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:46647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.75.2
--webserver_port=0
--tserver_master_addrs=127.3.75.62:33633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.2
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:03:24.849325 21121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:03:24.849555 21121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:03:24.854231 21127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:03:24.854271 21121 server_base.cc:1034] running on GCE node
W20260926 01:03:24.854188 21129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:24.854188 21126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:03:24.854545 21121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:03:24.854736 21121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:03:24.855866 21121 hybrid_clock.cc:648] HybridClock initialized: now 1790384604855861 us; error 30 us; skew 500 ppm
I20260926 01:03:24.857025 21121 webserver.cc:460] Webserver started at http://127.3.75.2:39371/ using document root <none> and password file <none>
I20260926 01:03:24.857276 21121 fs_manager.cc:359] Metadata directory not provided
I20260926 01:03:24.857321 21121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:03:24.857471 21121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:03:24.858531 21121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/ts-1/data/instance:
uuid: "86a3c26ae954422c85978c72c020f645"
format_stamp: "Formatted at 2026-09-26 01:03:24 on dist-test-slave-7v10"
server_key: "c081bd1171aadf77d710d21d1eb562d4"
server_key_iv: "1bbfc0a91a510d678bff04d213f6fded"
server_key_version: "encryptionkey@0"
I20260926 01:03:24.858932 21121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/ts-1/wal/instance:
uuid: "86a3c26ae954422c85978c72c020f645"
format_stamp: "Formatted at 2026-09-26 01:03:24 on dist-test-slave-7v10"
server_key: "c081bd1171aadf77d710d21d1eb562d4"
server_key_iv: "1bbfc0a91a510d678bff04d213f6fded"
server_key_version: "encryptionkey@0"
I20260926 01:03:24.860270 21121 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:03:24.860996 21135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:24.861276 21121 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:03:24.861343 21121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/ts-1/wal
uuid: "86a3c26ae954422c85978c72c020f645"
format_stamp: "Formatted at 2026-09-26 01:03:24 on dist-test-slave-7v10"
server_key: "c081bd1171aadf77d710d21d1eb562d4"
server_key_iv: "1bbfc0a91a510d678bff04d213f6fded"
server_key_version: "encryptionkey@0"
I20260926 01:03:24.861433 21121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:03:24.869261 21121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:03:24.869537 21121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:03:24.869688 21121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:03:24.869895 21121 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:03:24.870220 21121 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:03:24.870260 21121 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:24.870280 21121 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:03:24.870292 21121 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:24.875921 21121 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.2:45479
I20260926 01:03:24.876006 21249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.2:45479 every 8 connection(s)
I20260926 01:03:24.876278 21121 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
I20260926 01:03:24.879751 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 21121
I20260926 01:03:24.879848 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/ts-1/wal/instance
I20260926 01:03:24.880045 3372 external_mini_cluster.cc:1428] Setting key eaab973b5b80f55dfd3af837349f48fe
I20260926 01:03:24.881059 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.3:0
--local_ip_for_outbound_sockets=127.3.75.3
--webserver_interface=127.3.75.3
--webserver_port=0
--tserver_master_addrs=127.3.75.62:33633
--builtin_ntp_servers=127.3.75.20:46647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 01:03:24.881788 20968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:24.876974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:46639 (local address 127.3.75.62:33633)
0926 01:03:24.877040 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:24.877042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:24.877290 (+ 248us) server_negotiation.cc:407] Connection header received
0926 01:03:24.877759 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:24.877760 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:24.877769 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:24.877825 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:24.878430 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:24.878996 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:24.880263 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:24.880553 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:24.880624 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:24.881583 (+ 959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:24.881584 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:24.881584 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:03:24.881588 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:24.881600 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:03:24.881604 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:24.881647 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:24.881713 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 01:03:24.881732 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 01:03:24.882119 21252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:24.876947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:33633 (local address 127.3.75.2:46639)
0926 01:03:24.877147 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:24.877206 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:24.877682 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:24.877873 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:24.877875 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:24.877883 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:03:24.878357 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:24.878361 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:24.879072 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:24.879073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:24.880156 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:24.880159 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:24.880648 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:24.880650 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:24.880827 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:24.881426 (+ 599us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 01:03:24.881426 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:03:24.881465 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 01:03:24.881492 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:03:24.881649 (+ 157us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:03:24.881652 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:03:24.881655 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:03:24.881717 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 01:03:24.881819 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":116,"threads_started":1}
I20260926 01:03:24.882643 21250 heartbeater.cc:347] Connected to a master server at 127.3.75.62:33633
I20260926 01:03:24.882747 21250 heartbeater.cc:464] Registering TS with master...
I20260926 01:03:24.882958 21250 heartbeater.cc:511] Master 127.3.75.62:33633 requested a full tablet report, sending...
I20260926 01:03:24.883306 20928 ts_manager.cc:194] Registered new tserver with Master: 86a3c26ae954422c85978c72c020f645 (127.3.75.2:45479)
I20260926 01:03:24.883673 20928 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.75.2:46639
W20260926 01:03:24.959681 21253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:03:24.959863 21253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:03:24.959883 21253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:03:24.961012 21253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:03:24.961051 21253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:03:24.961102 21253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.3
I20260926 01:03:24.962387 21253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:46647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.75.3
--webserver_port=0
--tserver_master_addrs=127.3.75.62:33633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.3
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:03:24.962549 21253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:03:24.962839 21253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:03:24.967641 21259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:24.967664 21258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:24.967654 21261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:03:24.967897 21253 server_base.cc:1034] running on GCE node
I20260926 01:03:24.968065 21253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:03:24.968240 21253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:03:24.969374 21253 hybrid_clock.cc:648] HybridClock initialized: now 1790384604969360 us; error 31 us; skew 500 ppm
I20260926 01:03:24.970535 21253 webserver.cc:460] Webserver started at http://127.3.75.3:37831/ using document root <none> and password file <none>
I20260926 01:03:24.970755 21253 fs_manager.cc:359] Metadata directory not provided
I20260926 01:03:24.970796 21253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:03:24.970939 21253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:03:24.971906 21253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/ts-2/data/instance:
uuid: "502782bf168042ebb160e07f2fc0e8ee"
format_stamp: "Formatted at 2026-09-26 01:03:24 on dist-test-slave-7v10"
server_key: "44cc9d339056c88ddedd2c2791d072fb"
server_key_iv: "8dddb439dad06712baf27a455ca6420a"
server_key_version: "encryptionkey@0"
I20260926 01:03:24.972203 21253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/ts-2/wal/instance:
uuid: "502782bf168042ebb160e07f2fc0e8ee"
format_stamp: "Formatted at 2026-09-26 01:03:24 on dist-test-slave-7v10"
server_key: "44cc9d339056c88ddedd2c2791d072fb"
server_key_iv: "8dddb439dad06712baf27a455ca6420a"
server_key_version: "encryptionkey@0"
I20260926 01:03:24.973845 21253 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 01:03:24.974486 21267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:24.974651 21253 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 01:03:24.974720 21253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/ts-2/wal
uuid: "502782bf168042ebb160e07f2fc0e8ee"
format_stamp: "Formatted at 2026-09-26 01:03:24 on dist-test-slave-7v10"
server_key: "44cc9d339056c88ddedd2c2791d072fb"
server_key_iv: "8dddb439dad06712baf27a455ca6420a"
server_key_version: "encryptionkey@0"
I20260926 01:03:24.974768 21253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:03:24.985966 21253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:03:24.986245 21253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:03:24.986392 21253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:03:24.986613 21253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:03:24.986990 21253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:03:24.987046 21253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:24.987092 21253 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:03:24.987121 21253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:24.993209 21253 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.3:42005
I20260926 01:03:24.993242 21380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.3:42005 every 8 connection(s)
I20260926 01:03:24.993584 21253 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
I20260926 01:03:24.996331 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 21253
I20260926 01:03:24.996433 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/ts-2/wal/instance
I20260926 01:03:24.996618 3372 external_mini_cluster.cc:1428] Setting key 6ee6b719ba7ce2a7f4f7060dbbfa58d1
I20260926 01:03:24.999296 20968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:24.994294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.3:41415 (local address 127.3.75.62:33633)
0926 01:03:24.994336 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:24.994338 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:24.994591 (+ 253us) server_negotiation.cc:407] Connection header received
0926 01:03:24.995211 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:24.995213 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:24.995223 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:24.995288 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:24.995925 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:24.996618 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:24.997890 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:24.998072 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:24.998127 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:24.998964 (+ 837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:24.998966 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:24.998966 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:03:24.998971 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:24.998987 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:03:24.998991 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:24.999046 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:24.999166 (+ 120us) server_negotiation.cc:299] Negotiation successful
0926 01:03:24.999191 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 01:03:24.999555 21383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:24.994265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:33633 (local address 127.3.75.3:41415)
0926 01:03:24.994458 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:24.994501 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:24.995116 (+ 615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:24.995344 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:24.995347 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:24.995356 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:03:24.995854 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:24.995858 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:24.996732 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:24.996733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:24.997817 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:24.997820 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:24.998146 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:24.998147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:24.998254 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:24.998762 (+ 508us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 01:03:24.998762 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:03:24.998789 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 01:03:24.998811 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:03:24.999061 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:03:24.999067 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:03:24.999070 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:03:24.999148 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 01:03:24.999280 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":102,"threads_started":1}
I20260926 01:03:25.000079 21381 heartbeater.cc:347] Connected to a master server at 127.3.75.62:33633
I20260926 01:03:25.000175 21381 heartbeater.cc:464] Registering TS with master...
I20260926 01:03:25.000334 21381 heartbeater.cc:511] Master 127.3.75.62:33633 requested a full tablet report, sending...
I20260926 01:03:25.000705 20928 ts_manager.cc:194] Registered new tserver with Master: 502782bf168042ebb160e07f2fc0e8ee (127.3.75.3:42005)
I20260926 01:03:25.001019 20928 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.75.3:41415
I20260926 01:03:25.008368 3372 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 01:03:25.012421 20968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:25.009762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41224 (local address 127.3.75.62:33633)
0926 01:03:25.009824 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:25.009826 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:25.009896 (+ 70us) server_negotiation.cc:407] Connection header received
0926 01:03:25.009940 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:25.009941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:25.009948 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:25.010001 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:25.010160 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:25.010620 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:25.011378 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:25.011557 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:25.011609 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:25.012251 (+ 642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:25.012252 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:25.012252 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:03:25.012256 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:25.012266 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:03:25.012269 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:25.012305 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:25.012351 (+ 46us) server_negotiation.cc:299] Negotiation successful
0926 01:03:25.012367 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 01:03:25.013864 20928 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41224:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:03:25.015413 20928 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:03:25.021531 21390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:25.017494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.2:45479 (local address 127.0.0.1:36112)
0926 01:03:25.017666 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:25.017707 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:25.017769 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:25.018051 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:25.018053 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:25.018056 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:03:25.018118 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:25.018119 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:25.019272 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:25.019273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:25.020400 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:25.020402 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:25.021250 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:25.021251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:25.021323 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:25.021336 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 01:03:25.021411 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 01:03:25.021460 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":52,"threads_started":1}
I20260926 01:03:25.021544 21393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:25.017703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36112 (local address 127.3.75.2:45479)
0926 01:03:25.017894 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:25.017898 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:25.017909 (+ 11us) server_negotiation.cc:407] Connection header received
0926 01:03:25.017948 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:25.017950 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:25.017963 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:25.018026 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:03:25.018203 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:25.019183 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:25.020472 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:25.021158 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:25.021248 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:25.021343 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:25.021425 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 01:03:25.021460 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":82,"threads_started":1}
I20260926 01:03:25.022066 21389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:25.017494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.1:37353 (local address 127.0.0.1:41134)
0926 01:03:25.017796 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:25.017808 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:25.017864 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:25.018345 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:25.018346 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:25.018348 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:03:25.018389 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:25.018391 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:25.019713 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:25.019714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:25.020674 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:25.020675 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:25.021816 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:25.021817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:25.021883 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:25.021888 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 01:03:25.021936 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 01:03:25.021974 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":2,"thread_start_us":76,"threads_started":1}
I20260926 01:03:25.022154 21392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:25.017677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.3:42005 (local address 127.0.0.1:37772)
0926 01:03:25.018245 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:25.018255 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:25.018308 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:25.018554 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:25.018555 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:25.018557 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:03:25.018591 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:25.018593 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:25.019863 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:25.019864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:25.020901 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:25.020904 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:25.021912 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:25.021914 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:25.021984 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:25.021988 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:03:25.022051 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 01:03:25.022088 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":59,"threads_started":1}
I20260926 01:03:25.022367 21391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:25.017601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41134 (local address 127.3.75.1:37353)
0926 01:03:25.018119 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:25.018122 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:25.018129 (+ 7us) server_negotiation.cc:407] Connection header received
0926 01:03:25.018157 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:25.018158 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:25.018169 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:25.018262 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:03:25.018470 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:25.019620 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:25.020759 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:25.021435 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:25.021506 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:25.021586 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:25.022243 (+ 657us) server_negotiation.cc:299] Negotiation successful
0926 01:03:25.022288 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":100,"threads_started":1}
I20260926 01:03:25.022284 21183 tablet_service.cc:1467] Processing CreateTablet for tablet 109d2d4baadd459bb6547f10b9da307c (DEFAULT_TABLE table=test-table [id=b37d2b661fa64496a8c4805d2d08a792]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:03:25.022619 21394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:25.017853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37772 (local address 127.3.75.3:42005)
0926 01:03:25.018018 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:25.018023 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:25.018366 (+ 343us) server_negotiation.cc:407] Connection header received
0926 01:03:25.018412 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:25.018414 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:25.018428 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:25.018494 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:03:25.018671 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:25.019778 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:25.020988 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:25.021842 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:25.021912 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:25.021992 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:25.022059 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 01:03:25.022100 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":102,"threads_started":1}
I20260926 01:03:25.022694 21183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 109d2d4baadd459bb6547f10b9da307c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:03:25.023028 21051 tablet_service.cc:1467] Processing CreateTablet for tablet 109d2d4baadd459bb6547f10b9da307c (DEFAULT_TABLE table=test-table [id=b37d2b661fa64496a8c4805d2d08a792]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:03:25.023197 21315 tablet_service.cc:1467] Processing CreateTablet for tablet 109d2d4baadd459bb6547f10b9da307c (DEFAULT_TABLE table=test-table [id=b37d2b661fa64496a8c4805d2d08a792]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:03:25.023365 21051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 109d2d4baadd459bb6547f10b9da307c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:03:25.023494 21315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 109d2d4baadd459bb6547f10b9da307c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:03:25.025707 21395 tablet_bootstrap.cc:492] T 109d2d4baadd459bb6547f10b9da307c P 86a3c26ae954422c85978c72c020f645: Bootstrap starting.
I20260926 01:03:25.026312 21395 tablet_bootstrap.cc:654] T 109d2d4baadd459bb6547f10b9da307c P 86a3c26ae954422c85978c72c020f645: Neither blocks nor log segments found. Creating new log.
I20260926 01:03:25.026621 21397 tablet_bootstrap.cc:492] T 109d2d4baadd459bb6547f10b9da307c P 502782bf168042ebb160e07f2fc0e8ee: Bootstrap starting.
I20260926 01:03:25.026672 21395 log.cc:824] T 109d2d4baadd459bb6547f10b9da307c P 86a3c26ae954422c85978c72c020f645: Log is configured to *not* fsync() on all Append() calls
I20260926 01:03:25.026855 21396 tablet_bootstrap.cc:492] T 109d2d4baadd459bb6547f10b9da307c P d469a42ae6d446349aadea90ccceba57: Bootstrap starting.
I20260926 01:03:25.027529 21395 tablet_bootstrap.cc:492] T 109d2d4baadd459bb6547f10b9da307c P 86a3c26ae954422c85978c72c020f645: No bootstrap required, opened a new log
I20260926 01:03:25.027526 21396 tablet_bootstrap.cc:654] T 109d2d4baadd459bb6547f10b9da307c P d469a42ae6d446349aadea90ccceba57: Neither blocks nor log segments found. Creating new log.
I20260926 01:03:25.027601 21395 ts_tablet_manager.cc:1397] T 109d2d4baadd459bb6547f10b9da307c P 86a3c26ae954422c85978c72c020f645: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 01:03:25.027861 21396 log.cc:824] T 109d2d4baadd459bb6547f10b9da307c P d469a42ae6d446349aadea90ccceba57: Log is configured to *not* fsync() on all Append() calls
I20260926 01:03:25.028502 21396 tablet_bootstrap.cc:492] T 109d2d4baadd459bb6547f10b9da307c P d469a42ae6d446349aadea90ccceba57: No bootstrap required, opened a new log
I20260926 01:03:25.028585 21396 ts_tablet_manager.cc:1397] T 109d2d4baadd459bb6547f10b9da307c P d469a42ae6d446349aadea90ccceba57: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 01:03:25.028743 21397 tablet_bootstrap.cc:654] T 109d2d4baadd459bb6547f10b9da307c P 502782bf168042ebb160e07f2fc0e8ee: Neither blocks nor log segments found. Creating new log.
I20260926 01:03:25.029057 21397 log.cc:824] T 109d2d4baadd459bb6547f10b9da307c P 502782bf168042ebb160e07f2fc0e8ee: Log is configured to *not* fsync() on all Append() calls
I20260926 01:03:25.030118 21395 raft_consensus.cc:348] T 109d2d4baadd459bb6547f10b9da307c P 86a3c26ae954422c85978c72c020f645 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a3c26ae954422c85978c72c020f645" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 45479 } } peers { permanent_uuid: "502782bf168042ebb160e07f2fc0e8ee" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 42005 } } peers { permanent_uuid: "d469a42ae6d446349aadea90ccceba57" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37353 } }
I20260926 01:03:25.030248 21395 raft_consensus.cc:374] T 109d2d4baadd459bb6547f10b9da307c P 86a3c26ae954422c85978c72c020f645 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:03:25.030270 21395 raft_consensus.cc:729] T 109d2d4baadd459bb6547f10b9da307c P 86a3c26ae954422c85978c72c020f645 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86a3c26ae954422c85978c72c020f645, State: Initialized, Role: FOLLOWER
I20260926 01:03:25.030387 21395 consensus_queue.cc:260] T 109d2d4baadd459bb6547f10b9da307c P 86a3c26ae954422c85978c72c020f645 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a3c26ae954422c85978c72c020f645" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 45479 } } peers { permanent_uuid: "502782bf168042ebb160e07f2fc0e8ee" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 42005 } } peers { permanent_uuid: "d469a42ae6d446349aadea90ccceba57" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37353 } }
I20260926 01:03:25.030676 21395 ts_tablet_manager.cc:1428] T 109d2d4baadd459bb6547f10b9da307c P 86a3c26ae954422c85978c72c020f645: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 01:03:25.030795 21250 heartbeater.cc:503] Master 127.3.75.62:33633 was elected leader, sending a full tablet report...
I20260926 01:03:25.030939 21396 raft_consensus.cc:348] T 109d2d4baadd459bb6547f10b9da307c P d469a42ae6d446349aadea90ccceba57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a3c26ae954422c85978c72c020f645" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 45479 } } peers { permanent_uuid: "502782bf168042ebb160e07f2fc0e8ee" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 42005 } } peers { permanent_uuid: "d469a42ae6d446349aadea90ccceba57" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37353 } }
I20260926 01:03:25.031077 21396 raft_consensus.cc:374] T 109d2d4baadd459bb6547f10b9da307c P d469a42ae6d446349aadea90ccceba57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:03:25.031119 21396 raft_consensus.cc:729] T 109d2d4baadd459bb6547f10b9da307c P d469a42ae6d446349aadea90ccceba57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d469a42ae6d446349aadea90ccceba57, State: Initialized, Role: FOLLOWER
I20260926 01:03:25.031265 21396 consensus_queue.cc:260] T 109d2d4baadd459bb6547f10b9da307c P d469a42ae6d446349aadea90ccceba57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a3c26ae954422c85978c72c020f645" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 45479 } } peers { permanent_uuid: "502782bf168042ebb160e07f2fc0e8ee" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 42005 } } peers { permanent_uuid: "d469a42ae6d446349aadea90ccceba57" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37353 } }
I20260926 01:03:25.031440 21397 tablet_bootstrap.cc:492] T 109d2d4baadd459bb6547f10b9da307c P 502782bf168042ebb160e07f2fc0e8ee: No bootstrap required, opened a new log
I20260926 01:03:25.031507 21397 ts_tablet_manager.cc:1397] T 109d2d4baadd459bb6547f10b9da307c P 502782bf168042ebb160e07f2fc0e8ee: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260926 01:03:25.031529 21396 ts_tablet_manager.cc:1428] T 109d2d4baadd459bb6547f10b9da307c P d469a42ae6d446349aadea90ccceba57: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260926 01:03:25.031648 21118 heartbeater.cc:503] Master 127.3.75.62:33633 was elected leader, sending a full tablet report...
I20260926 01:03:25.033636 21397 raft_consensus.cc:348] T 109d2d4baadd459bb6547f10b9da307c P 502782bf168042ebb160e07f2fc0e8ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a3c26ae954422c85978c72c020f645" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 45479 } } peers { permanent_uuid: "502782bf168042ebb160e07f2fc0e8ee" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 42005 } } peers { permanent_uuid: "d469a42ae6d446349aadea90ccceba57" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37353 } }
I20260926 01:03:25.033769 21397 raft_consensus.cc:374] T 109d2d4baadd459bb6547f10b9da307c P 502782bf168042ebb160e07f2fc0e8ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:03:25.033789 21397 raft_consensus.cc:729] T 109d2d4baadd459bb6547f10b9da307c P 502782bf168042ebb160e07f2fc0e8ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 502782bf168042ebb160e07f2fc0e8ee, State: Initialized, Role: FOLLOWER
I20260926 01:03:25.033859 21397 consensus_queue.cc:260] T 109d2d4baadd459bb6547f10b9da307c P 502782bf168042ebb160e07f2fc0e8ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a3c26ae954422c85978c72c020f645" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 45479 } } peers { permanent_uuid: "502782bf168042ebb160e07f2fc0e8ee" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 42005 } } peers { permanent_uuid: "d469a42ae6d446349aadea90ccceba57" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37353 } }
I20260926 01:03:25.034024 21397 ts_tablet_manager.cc:1428] T 109d2d4baadd459bb6547f10b9da307c P 502782bf168042ebb160e07f2fc0e8ee: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 01:03:25.034103 21381 heartbeater.cc:503] Master 127.3.75.62:33633 was elected leader, sending a full tablet report...
I20260926 01:03:25.075987 21401 raft_consensus.cc:482] T 109d2d4baadd459bb6547f10b9da307c P 86a3c26ae954422c85978c72c020f645 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:03:25.076098 21401 raft_consensus.cc:504] T 109d2d4baadd459bb6547f10b9da307c P 86a3c26ae954422c85978c72c020f645 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a3c26ae954422c85978c72c020f645" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 45479 } } peers { permanent_uuid: "502782bf168042ebb160e07f2fc0e8ee" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 42005 } } peers { permanent_uuid: "d469a42ae6d446349aadea90ccceba57" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37353 } }
I20260926 01:03:25.076519 21401 leader_election.cc:288] T 109d2d4baadd459bb6547f10b9da307c P 86a3c26ae954422c85978c72c020f645 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 502782bf168042ebb160e07f2fc0e8ee (127.3.75.3:42005), d469a42ae6d446349aadea90ccceba57 (127.3.75.1:37353)
I20260926 01:03:25.079769 21252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:25.076764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.1:37353 (local address 127.3.75.2:49987)
0926 01:03:25.076831 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:25.076845 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:25.076894 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:25.077095 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:25.077096 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:25.077098 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:03:25.077208 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:25.077210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:25.077974 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:25.077975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:25.078803 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:25.078806 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:25.079533 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:25.079534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:25.079608 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:25.079611 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:03:25.079667 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 01:03:25.079714 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":39}
I20260926 01:03:25.079953 21394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:25.076836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:37509 (local address 127.3.75.3:42005)
0926 01:03:25.076881 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:25.076884 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:25.077109 (+ 225us) server_negotiation.cc:407] Connection header received
0926 01:03:25.077160 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:25.077161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:25.077170 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:25.077223 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:03:25.077382 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:25.077846 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:25.079125 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:25.079535 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:25.079603 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:25.079612 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:25.079836 (+ 224us) server_negotiation.cc:299] Negotiation successful
0926 01:03:25.079880 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 01:03:25.079973 21404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:25.076764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.3:42005 (local address 127.3.75.2:37509)
0926 01:03:25.077022 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:25.077033 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:25.077089 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:25.077234 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:25.077235 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:25.077236 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:03:25.077312 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:25.077314 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:25.077962 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:25.077963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:25.079024 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:25.079028 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:25.079632 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:25.079633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:25.079739 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:25.079743 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:03:25.079830 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 01:03:25.079881 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":17,"thread_start_us":150,"threads_started":1}
I20260926 01:03:25.079947 21391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:25.076836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:49987 (local address 127.3.75.1:37353)
0926 01:03:25.076898 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:25.076900 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:25.076909 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:03:25.076967 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:25.076969 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:25.076981 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:25.077052 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:03:25.077302 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:25.077834 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:25.078928 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:25.079440 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:25.079509 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:25.079520 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:25.079690 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 01:03:25.079738 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 01:03:25.080305 21071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "109d2d4baadd459bb6547f10b9da307c" candidate_uuid: "86a3c26ae954422c85978c72c020f645" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d469a42ae6d446349aadea90ccceba57" is_pre_election: true
I20260926 01:03:25.080330 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "109d2d4baadd459bb6547f10b9da307c" candidate_uuid: "86a3c26ae954422c85978c72c020f645" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "502782bf168042ebb160e07f2fc0e8ee" is_pre_election: true
I20260926 01:03:25.080488 21071 raft_consensus.cc:2454] T 109d2d4baadd459bb6547f10b9da307c P d469a42ae6d446349aadea90ccceba57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86a3c26ae954422c85978c72c020f645 in term 0.
I20260926 01:03:25.080488 21335 raft_consensus.cc:2454] T 109d2d4baadd459bb6547f10b9da307c P 502782bf168042ebb160e07f2fc0e8ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86a3c26ae954422c85978c72c020f645 in term 0.
I20260926 01:03:25.080740 21137 leader_election.cc:302] T 109d2d4baadd459bb6547f10b9da307c P 86a3c26ae954422c85978c72c020f645 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 502782bf168042ebb160e07f2fc0e8ee, 86a3c26ae954422c85978c72c020f645; no voters:
I20260926 01:03:25.080912 21401 raft_consensus.cc:2781] T 109d2d4baadd459bb6547f10b9da307c P 86a3c26ae954422c85978c72c020f645 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:03:25.080981 21401 raft_consensus.cc:482] T 109d2d4baadd459bb6547f10b9da307c P 86a3c26ae954422c85978c72c020f645 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:03:25.081020 21401 raft_consensus.cc:3037] T 109d2d4baadd459bb6547f10b9da307c P 86a3c26ae954422c85978c72c020f645 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:03:25.081797 21401 raft_consensus.cc:504] T 109d2d4baadd459bb6547f10b9da307c P 86a3c26ae954422c85978c72c020f645 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a3c26ae954422c85978c72c020f645" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 45479 } } peers { permanent_uuid: "502782bf168042ebb160e07f2fc0e8ee" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 42005 } } peers { permanent_uuid: "d469a42ae6d446349aadea90ccceba57" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37353 } }
I20260926 01:03:25.081969 21401 leader_election.cc:288] T 109d2d4baadd459bb6547f10b9da307c P 86a3c26ae954422c85978c72c020f645 [CANDIDATE]: Term 1 election: Requested vote from peers 502782bf168042ebb160e07f2fc0e8ee (127.3.75.3:42005), d469a42ae6d446349aadea90ccceba57 (127.3.75.1:37353)
I20260926 01:03:25.082170 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "109d2d4baadd459bb6547f10b9da307c" candidate_uuid: "86a3c26ae954422c85978c72c020f645" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "502782bf168042ebb160e07f2fc0e8ee"
I20260926 01:03:25.082187 21071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "109d2d4baadd459bb6547f10b9da307c" candidate_uuid: "86a3c26ae954422c85978c72c020f645" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d469a42ae6d446349aadea90ccceba57"
I20260926 01:03:25.082270 21335 raft_consensus.cc:3037] T 109d2d4baadd459bb6547f10b9da307c P 502782bf168042ebb160e07f2fc0e8ee [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:03:25.082271 21071 raft_consensus.cc:3037] T 109d2d4baadd459bb6547f10b9da307c P d469a42ae6d446349aadea90ccceba57 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:03:25.083334 21071 raft_consensus.cc:2454] T 109d2d4baadd459bb6547f10b9da307c P d469a42ae6d446349aadea90ccceba57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86a3c26ae954422c85978c72c020f645 in term 1.
I20260926 01:03:25.083334 21335 raft_consensus.cc:2454] T 109d2d4baadd459bb6547f10b9da307c P 502782bf168042ebb160e07f2fc0e8ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86a3c26ae954422c85978c72c020f645 in term 1.
I20260926 01:03:25.083546 21139 leader_election.cc:302] T 109d2d4baadd459bb6547f10b9da307c P 86a3c26ae954422c85978c72c020f645 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86a3c26ae954422c85978c72c020f645, d469a42ae6d446349aadea90ccceba57; no voters:
I20260926 01:03:25.083671 21401 raft_consensus.cc:2781] T 109d2d4baadd459bb6547f10b9da307c P 86a3c26ae954422c85978c72c020f645 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:03:25.083861 21401 raft_consensus.cc:686] T 109d2d4baadd459bb6547f10b9da307c P 86a3c26ae954422c85978c72c020f645 [term 1 LEADER]: Becoming Leader. State: Replica: 86a3c26ae954422c85978c72c020f645, State: Running, Role: LEADER
I20260926 01:03:25.083972 21401 consensus_queue.cc:237] T 109d2d4baadd459bb6547f10b9da307c P 86a3c26ae954422c85978c72c020f645 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a3c26ae954422c85978c72c020f645" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 45479 } } peers { permanent_uuid: "502782bf168042ebb160e07f2fc0e8ee" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 42005 } } peers { permanent_uuid: "d469a42ae6d446349aadea90ccceba57" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37353 } }
I20260926 01:03:25.084824 20928 catalog_manager.cc:5495] T 109d2d4baadd459bb6547f10b9da307c P 86a3c26ae954422c85978c72c020f645 reported cstate change: term changed from 0 to 1, leader changed from <none> to 86a3c26ae954422c85978c72c020f645 (127.3.75.2). New cstate: current_term: 1 leader_uuid: "86a3c26ae954422c85978c72c020f645" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a3c26ae954422c85978c72c020f645" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 45479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "502782bf168042ebb160e07f2fc0e8ee" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 42005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d469a42ae6d446349aadea90ccceba57" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37353 } health_report { overall_health: UNKNOWN } } }
I20260926 01:03:25.094640 21393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:25.092255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36118 (local address 127.3.75.2:45479)
0926 01:03:25.092324 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:25.092326 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:25.092336 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:03:25.092409 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:25.092409 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:25.092435 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:25.092514 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:03:25.092651 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:25.093107 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:25.093859 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:25.094055 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:25.094129 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:25.094305 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:03:25.094400 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:03:25.094494 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:25.094520 (+ 26us) server_negotiation.cc:299] Negotiation successful
0926 01:03:25.094537 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 01:03:25.096400 21335 raft_consensus.cc:1261] T 109d2d4baadd459bb6547f10b9da307c P 502782bf168042ebb160e07f2fc0e8ee [term 1 FOLLOWER]: Refusing update from remote peer 86a3c26ae954422c85978c72c020f645: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:03:25.096400 21071 raft_consensus.cc:1261] T 109d2d4baadd459bb6547f10b9da307c P d469a42ae6d446349aadea90ccceba57 [term 1 FOLLOWER]: Refusing update from remote peer 86a3c26ae954422c85978c72c020f645: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:03:25.096804 21401 consensus_queue.cc:1035] T 109d2d4baadd459bb6547f10b9da307c P 86a3c26ae954422c85978c72c020f645 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d469a42ae6d446349aadea90ccceba57" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 37353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:03:25.096964 21405 consensus_queue.cc:1035] T 109d2d4baadd459bb6547f10b9da307c P 86a3c26ae954422c85978c72c020f645 [LEADER]: Connected to new peer: Peer: permanent_uuid: "502782bf168042ebb160e07f2fc0e8ee" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 42005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:03:25.099325 21409 mvcc.cc:205] Tried to move back new op lower bound from 7333415342472224768 to 7333415342424637440. Current Snapshot: MvccSnapshot[applied={T|T < 7333415342472224768}]
I20260926 01:03:25.099967 21410 mvcc.cc:205] Tried to move back new op lower bound from 7333415342472224768 to 7333415342424637440. Current Snapshot: MvccSnapshot[applied={T|T < 7333415342472224768}]
I20260926 01:03:25.101275 20926 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:03:25.101363 20926 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:03:25.102100 20926 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b723884a155a447e866a20a6cda7f211: Sending DeleteTablet for 3 replicas of tablet 109d2d4baadd459bb6547f10b9da307c
I20260926 01:03:25.102403 21183 tablet_service.cc:1514] Processing DeleteTablet for tablet 109d2d4baadd459bb6547f10b9da307c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:03:25 UTC) from {username='slave'} at 127.0.0.1:36112
I20260926 01:03:25.102437 21051 tablet_service.cc:1514] Processing DeleteTablet for tablet 109d2d4baadd459bb6547f10b9da307c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:03:25 UTC) from {username='slave'} at 127.0.0.1:41134
I20260926 01:03:25.102670 21418 tablet_replica.cc:330] T 109d2d4baadd459bb6547f10b9da307c P 86a3c26ae954422c85978c72c020f645: stopping tablet replica
I20260926 01:03:25.102666 21315 tablet_service.cc:1514] Processing DeleteTablet for tablet 109d2d4baadd459bb6547f10b9da307c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:03:25 UTC) from {username='slave'} at 127.0.0.1:37772
I20260926 01:03:25.102775 21418 raft_consensus.cc:2229] T 109d2d4baadd459bb6547f10b9da307c P 86a3c26ae954422c85978c72c020f645 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:03:25.102927 21418 raft_consensus.cc:2258] T 109d2d4baadd459bb6547f10b9da307c P 86a3c26ae954422c85978c72c020f645 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:03:25.102936 21420 tablet_replica.cc:330] T 109d2d4baadd459bb6547f10b9da307c P 502782bf168042ebb160e07f2fc0e8ee: stopping tablet replica
I20260926 01:03:25.103111 21420 raft_consensus.cc:2229] T 109d2d4baadd459bb6547f10b9da307c P 502782bf168042ebb160e07f2fc0e8ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:03:25.103248 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 20989
I20260926 01:03:25.104187 21420 raft_consensus.cc:2258] T 109d2d4baadd459bb6547f10b9da307c P 502782bf168042ebb160e07f2fc0e8ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:03:25.104697 21420 ts_tablet_manager.cc:1905] T 109d2d4baadd459bb6547f10b9da307c P 502782bf168042ebb160e07f2fc0e8ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:03:25.104758 21418 ts_tablet_manager.cc:1905] T 109d2d4baadd459bb6547f10b9da307c P 86a3c26ae954422c85978c72c020f645: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:03:25.106640 21420 ts_tablet_manager.cc:1918] T 109d2d4baadd459bb6547f10b9da307c P 502782bf168042ebb160e07f2fc0e8ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:03:25.106789 21420 log.cc:1196] T 109d2d4baadd459bb6547f10b9da307c P 502782bf168042ebb160e07f2fc0e8ee: Deleting WAL directory at /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/ts-2/wal/wals/109d2d4baadd459bb6547f10b9da307c
I20260926 01:03:25.106894 21418 ts_tablet_manager.cc:1918] T 109d2d4baadd459bb6547f10b9da307c P 86a3c26ae954422c85978c72c020f645: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:03:25.106973 21418 log.cc:1196] T 109d2d4baadd459bb6547f10b9da307c P 86a3c26ae954422c85978c72c020f645: Deleting WAL directory at /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384372073861-3372-0/minicluster-data/ts-1/wal/wals/109d2d4baadd459bb6547f10b9da307c
I20260926 01:03:25.107261 21420 ts_tablet_manager.cc:1939] T 109d2d4baadd459bb6547f10b9da307c P 502782bf168042ebb160e07f2fc0e8ee: Deleting consensus metadata
I20260926 01:03:25.107288 21418 ts_tablet_manager.cc:1939] T 109d2d4baadd459bb6547f10b9da307c P 86a3c26ae954422c85978c72c020f645: Deleting consensus metadata
I20260926 01:03:25.107722 20913 catalog_manager.cc:4841] TS 86a3c26ae954422c85978c72c020f645 (127.3.75.2:45479): tablet 109d2d4baadd459bb6547f10b9da307c (table test-table [id=b37d2b661fa64496a8c4805d2d08a792]) successfully deleted
I20260926 01:03:25.107822 20913 catalog_manager.cc:4841] TS 502782bf168042ebb160e07f2fc0e8ee (127.3.75.3:42005): tablet 109d2d4baadd459bb6547f10b9da307c (table test-table [id=b37d2b661fa64496a8c4805d2d08a792]) successfully deleted
W20260926 01:03:25.109642 20915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.75.1:37353 (error 108)
I20260926 01:03:25.110158 21389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:25.110011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.1:37353 (local address 127.0.0.1:41148)
0926 01:03:25.110074 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:25.110119 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.1:37353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 01:03:25.110183 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 21121
W20260926 01:03:25.111020 20915 catalog_manager.cc:4584] TS d469a42ae6d446349aadea90ccceba57 (127.3.75.1:37353): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 109d2d4baadd459bb6547f10b9da307c: Network error: Client connection negotiation failed: client connection to 127.3.75.1:37353: connect: Connection refused (error 111)
I20260926 01:03:25.115339 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 21253
I20260926 01:03:25.119967 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 20897
2026-09-26T01:03:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (643 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:03:25 dist-test-slave-7v10 krb5kdc[21424](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:03:25 dist-test-slave-7v10 krb5kdc[21424](info): set up 2 sockets
Sep 26 01:03:25 dist-test-slave-7v10 krb5kdc[21424](info): commencing operation
krb5kdc: starting...
W20260926 01:03:26.145569 3372 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Sep 26 01:03:26 dist-test-slave-7v10 krb5kdc[21424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:03:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:03:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.75.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab.
Entry for principal kudu/127.3.75.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab.
I20260926 01:03:26.261873 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:41399
--webserver_interface=127.3.75.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:36235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.75.62:41399
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:03:26.342021 21437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:03:26.342253 21437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:03:26.342288 21437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:03:26.343423 21437 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:03:26.343484 21437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:03:26.343508 21437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:03:26.343528 21437 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:03:26.343565 21437 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:03:26.344977 21437 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:36235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.75.62:41399
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:41399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:03:26.345240 21437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:03:26.345486 21437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:03:26.350203 21443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:26.350257 21446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:26.350203 21444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:03:26.350327 21437 server_base.cc:1034] running on GCE node
I20260926 01:03:26.350628 21437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:03:26.350975 21437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:03:26.352126 21437 hybrid_clock.cc:648] HybridClock initialized: now 1790384606352120 us; error 37 us; skew 500 ppm
Sep 26 01:03:26 dist-test-slave-7v10 krb5kdc[21424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384606, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:03:26.354417 21437 init.cc:377] Logged in from keytab as kudu/127.3.75.62@KRBTEST.COM (short username kudu)
I20260926 01:03:26.354851 21437 webserver.cc:460] Webserver started at http://127.3.75.62:36649/ using document root <none> and password file <none>
I20260926 01:03:26.355135 21437 fs_manager.cc:359] Metadata directory not provided
I20260926 01:03:26.355211 21437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:03:26.355345 21437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:03:26.356428 21437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/master-0/data/instance:
uuid: "c54c1730e2c644cba915b8f409d4c659"
format_stamp: "Formatted at 2026-09-26 01:03:26 on dist-test-slave-7v10"
server_key: "a4eb841c54ab08fd8f0f7617b110259b"
server_key_iv: "c8e13079bf7488310110ccb0f0e6e7b9"
server_key_version: "encryptionkey@0"
I20260926 01:03:26.356784 21437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/master-0/wal/instance:
uuid: "c54c1730e2c644cba915b8f409d4c659"
format_stamp: "Formatted at 2026-09-26 01:03:26 on dist-test-slave-7v10"
server_key: "a4eb841c54ab08fd8f0f7617b110259b"
server_key_iv: "c8e13079bf7488310110ccb0f0e6e7b9"
server_key_version: "encryptionkey@0"
I20260926 01:03:26.358039 21437 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:03:26.358702 21453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:26.358865 21437 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:03:26.358992 21437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/master-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/master-0/wal
uuid: "c54c1730e2c644cba915b8f409d4c659"
format_stamp: "Formatted at 2026-09-26 01:03:26 on dist-test-slave-7v10"
server_key: "a4eb841c54ab08fd8f0f7617b110259b"
server_key_iv: "c8e13079bf7488310110ccb0f0e6e7b9"
server_key_version: "encryptionkey@0"
I20260926 01:03:26.359068 21437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:03:26.380003 21437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:03:26.381763 21437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:03:26.381918 21437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:03:26.385694 21437 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.62:41399
I20260926 01:03:26.385731 21505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.62:41399 every 8 connection(s)
I20260926 01:03:26.386088 21437 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
I20260926 01:03:26.386394 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 21437
I20260926 01:03:26.386536 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/master-0/wal/instance
I20260926 01:03:26.386765 3372 external_mini_cluster.cc:1428] Setting key 8ec1ae367e8122d7a5255c3d9b3a0fb1
I20260926 01:03:26.387167 21506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:03:26.390329 21506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c54c1730e2c644cba915b8f409d4c659: Bootstrap starting.
I20260926 01:03:26.391047 21506 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c54c1730e2c644cba915b8f409d4c659: Neither blocks nor log segments found. Creating new log.
I20260926 01:03:26.391345 21506 log.cc:824] T 00000000000000000000000000000000 P c54c1730e2c644cba915b8f409d4c659: Log is configured to *not* fsync() on all Append() calls
I20260926 01:03:26.392141 21506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c54c1730e2c644cba915b8f409d4c659: No bootstrap required, opened a new log
Sep 26 01:03:26 dist-test-slave-7v10 krb5kdc[21424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 01:03:26.394537 21506 raft_consensus.cc:348] T 00000000000000000000000000000000 P c54c1730e2c644cba915b8f409d4c659 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c54c1730e2c644cba915b8f409d4c659" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 41399 } }
I20260926 01:03:26.394723 21506 raft_consensus.cc:374] T 00000000000000000000000000000000 P c54c1730e2c644cba915b8f409d4c659 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:03:26.394765 21506 raft_consensus.cc:729] T 00000000000000000000000000000000 P c54c1730e2c644cba915b8f409d4c659 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c54c1730e2c644cba915b8f409d4c659, State: Initialized, Role: FOLLOWER
I20260926 01:03:26.394923 21506 consensus_queue.cc:260] T 00000000000000000000000000000000 P c54c1730e2c644cba915b8f409d4c659 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c54c1730e2c644cba915b8f409d4c659" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 41399 } }
I20260926 01:03:26.395001 21506 raft_consensus.cc:388] T 00000000000000000000000000000000 P c54c1730e2c644cba915b8f409d4c659 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:03:26.395022 21506 raft_consensus.cc:482] T 00000000000000000000000000000000 P c54c1730e2c644cba915b8f409d4c659 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:03:26.395076 21506 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c54c1730e2c644cba915b8f409d4c659 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:03:26.395539 21506 raft_consensus.cc:504] T 00000000000000000000000000000000 P c54c1730e2c644cba915b8f409d4c659 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c54c1730e2c644cba915b8f409d4c659" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 41399 } }
I20260926 01:03:26.395644 21506 leader_election.cc:302] T 00000000000000000000000000000000 P c54c1730e2c644cba915b8f409d4c659 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c54c1730e2c644cba915b8f409d4c659; no voters:
I20260926 01:03:26.395823 21506 leader_election.cc:288] T 00000000000000000000000000000000 P c54c1730e2c644cba915b8f409d4c659 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:03:26.395989 21512 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c54c1730e2c644cba915b8f409d4c659 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:03:26.396095 21506 sys_catalog.cc:564] T 00000000000000000000000000000000 P c54c1730e2c644cba915b8f409d4c659 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:03:26.396148 21512 raft_consensus.cc:686] T 00000000000000000000000000000000 P c54c1730e2c644cba915b8f409d4c659 [term 1 LEADER]: Becoming Leader. State: Replica: c54c1730e2c644cba915b8f409d4c659, State: Running, Role: LEADER
I20260926 01:03:26.396494 21512 consensus_queue.cc:237] T 00000000000000000000000000000000 P c54c1730e2c644cba915b8f409d4c659 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c54c1730e2c644cba915b8f409d4c659" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 41399 } }
I20260926 01:03:26.396857 21512 sys_catalog.cc:455] T 00000000000000000000000000000000 P c54c1730e2c644cba915b8f409d4c659 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c54c1730e2c644cba915b8f409d4c659. Latest consensus state: current_term: 1 leader_uuid: "c54c1730e2c644cba915b8f409d4c659" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c54c1730e2c644cba915b8f409d4c659" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 41399 } } }
I20260926 01:03:26.396961 21512 sys_catalog.cc:458] T 00000000000000000000000000000000 P c54c1730e2c644cba915b8f409d4c659 [sys.catalog]: This master's current role is: LEADER
I20260926 01:03:26.397197 21521 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:03:26.397715 21521 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:03:26.397759 21509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:26.387312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44386 (local address 127.3.75.62:41399)
0926 01:03:26.387482 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:26.387495 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:26.387519 (+ 24us) server_negotiation.cc:407] Connection header received
0926 01:03:26.387934 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:26.387941 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:26.387985 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:26.388098 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:26.388859 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:26.389715 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:26.390783 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:26.391252 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:26.391332 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:26.393931 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:26.393937 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:26.393937 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:03:26.393950 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:26.395762 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:26.396146 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:26.396149 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:26.396149 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:26.396268 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:26.396567 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:26.396568 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:26.396569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:26.396898 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:26.397023 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:26.397335 (+ 312us) server_negotiation.cc:299] Negotiation successful
0926 01:03:26.397497 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":91,"threads_started":1}
I20260926 01:03:26.398476 21513 sys_catalog.cc:455] T 00000000000000000000000000000000 P c54c1730e2c644cba915b8f409d4c659 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c54c1730e2c644cba915b8f409d4c659" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c54c1730e2c644cba915b8f409d4c659" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 41399 } } }
I20260926 01:03:26.398602 21513 sys_catalog.cc:458] T 00000000000000000000000000000000 P c54c1730e2c644cba915b8f409d4c659 [sys.catalog]: This master's current role is: LEADER
I20260926 01:03:26.399516 21521 catalog_manager.cc:1347] Generated new cluster ID: fcffcebfad044380bcb6e47b98dafedb
I20260926 01:03:26.399572 21521 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:03:26.417455 21521 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:03:26.418139 21521 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:03:26.424795 21521 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c54c1730e2c644cba915b8f409d4c659: Generated new TSK 0
I20260926 01:03:26.424957 21521 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.75.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab.
Entry for principal kudu/127.3.75.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab.
I20260926 01:03:26.453063 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.1:0
--local_ip_for_outbound_sockets=127.3.75.1
--webserver_interface=127.3.75.1
--webserver_port=0
--tserver_master_addrs=127.3.75.62:41399
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab
--principal=kudu/127.3.75.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:36235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:03:26.531836 21533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:03:26.532029 21533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:03:26.532051 21533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:03:26.533169 21533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:03:26.533210 21533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:03:26.533258 21533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.1
I20260926 01:03:26.534672 21533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:36235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab
--principal=kudu/127.3.75.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.75.1
--webserver_port=0
--tserver_master_addrs=127.3.75.62:41399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.1
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:03:26.534929 21533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:03:26.535140 21533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:03:26.539907 21539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:26.539873 21541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:26.539873 21538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:03:26.540196 21533 server_base.cc:1034] running on GCE node
I20260926 01:03:26.540437 21533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:03:26.540678 21533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:03:26.541810 21533 hybrid_clock.cc:648] HybridClock initialized: now 1790384606541800 us; error 26 us; skew 500 ppm
Sep 26 01:03:26 dist-test-slave-7v10 krb5kdc[21424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384606, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:03:26.544238 21533 init.cc:377] Logged in from keytab as kudu/127.3.75.1@KRBTEST.COM (short username kudu)
I20260926 01:03:26.544708 21533 webserver.cc:460] Webserver started at http://127.3.75.1:45843/ using document root <none> and password file <none>
I20260926 01:03:26.544907 21533 fs_manager.cc:359] Metadata directory not provided
I20260926 01:03:26.544950 21533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:03:26.545081 21533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:03:26.546032 21533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/ts-0/data/instance:
uuid: "5f93e7d57c6a4c41a0d9bc91196e7887"
format_stamp: "Formatted at 2026-09-26 01:03:26 on dist-test-slave-7v10"
server_key: "d3bae8757f0f2c472cbad48774990356"
server_key_iv: "713eea8e524d5ca1ca029e9fdedd9a13"
server_key_version: "encryptionkey@0"
I20260926 01:03:26.546360 21533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/ts-0/wal/instance:
uuid: "5f93e7d57c6a4c41a0d9bc91196e7887"
format_stamp: "Formatted at 2026-09-26 01:03:26 on dist-test-slave-7v10"
server_key: "d3bae8757f0f2c472cbad48774990356"
server_key_iv: "713eea8e524d5ca1ca029e9fdedd9a13"
server_key_version: "encryptionkey@0"
I20260926 01:03:26.547684 21533 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:03:26.548368 21548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:26.548549 21533 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:03:26.548632 21533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/ts-0/wal
uuid: "5f93e7d57c6a4c41a0d9bc91196e7887"
format_stamp: "Formatted at 2026-09-26 01:03:26 on dist-test-slave-7v10"
server_key: "d3bae8757f0f2c472cbad48774990356"
server_key_iv: "713eea8e524d5ca1ca029e9fdedd9a13"
server_key_version: "encryptionkey@0"
I20260926 01:03:26.548712 21533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:03:26.581127 21533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:03:26.582856 21533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:03:26.583031 21533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:03:26.583276 21533 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:03:26.583654 21533 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:03:26.583689 21533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:26.583727 21533 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:03:26.583755 21533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:26.589438 21533 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.1:42299
I20260926 01:03:26.589489 21662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.1:42299 every 8 connection(s)
I20260926 01:03:26.589805 21533 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
Sep 26 01:03:26 dist-test-slave-7v10 krb5kdc[21424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384606, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.1@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 01:03:26.597813 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 21533
I20260926 01:03:26.597955 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/ts-0/wal/instance
I20260926 01:03:26.598206 3372 external_mini_cluster.cc:1428] Setting key f990c25f5525066d0690fead5eb3297c
I20260926 01:03:26.599238 21509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:26.590489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:54555 (local address 127.3.75.62:41399)
0926 01:03:26.590548 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:26.590550 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:26.590752 (+ 202us) server_negotiation.cc:407] Connection header received
0926 01:03:26.591301 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:26.591302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:26.591313 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:26.591370 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:26.592315 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:26.592809 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:26.593766 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:26.593952 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:26.594004 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:26.596278 (+ 2274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:26.596279 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:26.596279 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:03:26.596286 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:26.597663 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:26.598198 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:26.598199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:26.598200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:26.598266 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:26.598536 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:26.598537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:26.598538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:26.598652 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:26.598729 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:26.599053 (+ 324us) server_negotiation.cc:299] Negotiation successful
0926 01:03:26.599132 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 01:03:26.599454 21665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:26.590450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:41399 (local address 127.3.75.1:54555)
0926 01:03:26.590613 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:26.590656 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:26.591211 (+ 555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:26.591434 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:26.591437 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:26.592029 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:03:26.592231 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:26.592237 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:26.592888 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:26.592889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:26.593691 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:26.593693 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:26.594023 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:26.594024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:26.594124 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:26.594648 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:03:26.594656 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:03:26.596187 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:03:26.597745 (+ 1558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:03:26.597753 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:03:26.597756 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:03:26.598128 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:03:26.598354 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:03:26.598355 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:03:26.598356 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:03:26.598470 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:03:26.598732 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:03:26.598739 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:03:26.598912 (+ 173us) client_negotiation.cc:769] Sending connection context
0926 01:03:26.599031 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 01:03:26.599197 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":86,"threads_started":1}
I20260926 01:03:26.599896 21663 heartbeater.cc:347] Connected to a master server at 127.3.75.62:41399
I20260926 01:03:26.599985 21663 heartbeater.cc:464] Registering TS with master...
I20260926 01:03:26.600172 21663 heartbeater.cc:511] Master 127.3.75.62:41399 requested a full tablet report, sending...
I20260926 01:03:26.600682 21470 ts_manager.cc:194] Registered new tserver with Master: 5f93e7d57c6a4c41a0d9bc91196e7887 (127.3.75.1:42299)
I20260926 01:03:26.601280 21470 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.75.1@KRBTEST.COM'} at 127.3.75.1:54555
WARNING: no policy specified for kudu/127.3.75.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab.
Entry for principal kudu/127.3.75.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab.
I20260926 01:03:26.621939 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.2:0
--local_ip_for_outbound_sockets=127.3.75.2
--webserver_interface=127.3.75.2
--webserver_port=0
--tserver_master_addrs=127.3.75.62:41399
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab
--principal=kudu/127.3.75.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:36235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:03:26.701052 21668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:03:26.701272 21668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:03:26.701306 21668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:03:26.702417 21668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:03:26.702479 21668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:03:26.702522 21668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.2
I20260926 01:03:26.703969 21668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:36235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab
--principal=kudu/127.3.75.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.75.2
--webserver_port=0
--tserver_master_addrs=127.3.75.62:41399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.2
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:03:26.704221 21668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:03:26.704448 21668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:03:26.709375 21676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:03:26.709426 21668 server_base.cc:1034] running on GCE node
W20260926 01:03:26.709420 21674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:26.709537 21673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:03:26.709753 21668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:03:26.710000 21668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:03:26.711150 21668 hybrid_clock.cc:648] HybridClock initialized: now 1790384606711137 us; error 38 us; skew 500 ppm
Sep 26 01:03:26 dist-test-slave-7v10 krb5kdc[21424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384606, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:03:26.713519 21668 init.cc:377] Logged in from keytab as kudu/127.3.75.2@KRBTEST.COM (short username kudu)
I20260926 01:03:26.713924 21668 webserver.cc:460] Webserver started at http://127.3.75.2:33659/ using document root <none> and password file <none>
I20260926 01:03:26.714148 21668 fs_manager.cc:359] Metadata directory not provided
I20260926 01:03:26.714213 21668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:03:26.714360 21668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:03:26.715391 21668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/ts-1/data/instance:
uuid: "be9b12ad8128457bb6267680270da159"
format_stamp: "Formatted at 2026-09-26 01:03:26 on dist-test-slave-7v10"
server_key: "6b20cc9cb7177fecabbfb27bea8136c6"
server_key_iv: "726b7867d671cdaa8ccf38e00f626283"
server_key_version: "encryptionkey@0"
I20260926 01:03:26.715719 21668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/ts-1/wal/instance:
uuid: "be9b12ad8128457bb6267680270da159"
format_stamp: "Formatted at 2026-09-26 01:03:26 on dist-test-slave-7v10"
server_key: "6b20cc9cb7177fecabbfb27bea8136c6"
server_key_iv: "726b7867d671cdaa8ccf38e00f626283"
server_key_version: "encryptionkey@0"
I20260926 01:03:26.716952 21668 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:03:26.717650 21683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:26.717813 21668 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:03:26.717909 21668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/ts-1/wal
uuid: "be9b12ad8128457bb6267680270da159"
format_stamp: "Formatted at 2026-09-26 01:03:26 on dist-test-slave-7v10"
server_key: "6b20cc9cb7177fecabbfb27bea8136c6"
server_key_iv: "726b7867d671cdaa8ccf38e00f626283"
server_key_version: "encryptionkey@0"
I20260926 01:03:26.717978 21668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:03:26.726624 21668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:03:26.728376 21668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:03:26.728526 21668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:03:26.728726 21668 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:03:26.729012 21668 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:03:26.729061 21668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:26.729101 21668 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:03:26.729130 21668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:26.734666 21668 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.2:42577
I20260926 01:03:26.734714 21797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.2:42577 every 8 connection(s)
I20260926 01:03:26.735044 21668 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
I20260926 01:03:26.736164 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 21668
I20260926 01:03:26.736304 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/ts-1/wal/instance
I20260926 01:03:26.736511 3372 external_mini_cluster.cc:1428] Setting key 410ae6b69d3d55c681959851c0ab1cec
Sep 26 01:03:26 dist-test-slave-7v10 krb5kdc[21424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384606, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.2@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.75.3@KRBTEST.COM; defaulting to no policy
I20260926 01:03:26.745353 21509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:26.735700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:34415 (local address 127.3.75.62:41399)
0926 01:03:26.735767 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:26.735769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:26.735914 (+ 145us) server_negotiation.cc:407] Connection header received
0926 01:03:26.736865 (+ 951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:26.736866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:26.736879 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:26.736935 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:26.738165 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:26.738637 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:26.739817 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:26.740012 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:26.740091 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:26.742543 (+ 2452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:26.742545 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:26.742545 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:03:26.742551 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:26.743909 (+ 1358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:26.744401 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:26.744401 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:26.744402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:26.744448 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:26.744725 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:26.744726 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:26.744726 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:26.744820 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:26.744865 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:26.745225 (+ 360us) server_negotiation.cc:299] Negotiation successful
0926 01:03:26.745291 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 01:03:26.745599 21800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:26.735651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:41399 (local address 127.3.75.2:34415)
0926 01:03:26.735799 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:26.735838 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:26.736757 (+ 919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:26.736992 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:26.736996 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:26.737807 (+ 811us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:03:26.738078 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:26.738084 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:26.738746 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:26.738747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:26.739707 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:26.739710 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:26.740099 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:26.740099 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:26.740229 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:26.740726 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:03:26.740735 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:03:26.742455 (+ 1720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:03:26.743991 (+ 1536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:03:26.743998 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:03:26.744001 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:03:26.744325 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:03:26.744522 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:03:26.744522 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:03:26.744523 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:03:26.744658 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:03:26.744878 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:03:26.744884 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:03:26.745073 (+ 189us) client_negotiation.cc:769] Sending connection context
0926 01:03:26.745210 (+ 137us) client_negotiation.cc:240] Negotiation successful
0926 01:03:26.745365 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":83,"threads_started":1}
I20260926 01:03:26.746135 21798 heartbeater.cc:347] Connected to a master server at 127.3.75.62:41399
I20260926 01:03:26.746217 21798 heartbeater.cc:464] Registering TS with master...
I20260926 01:03:26.746347 21798 heartbeater.cc:511] Master 127.3.75.62:41399 requested a full tablet report, sending...
I20260926 01:03:26.746686 21470 ts_manager.cc:194] Registered new tserver with Master: be9b12ad8128457bb6267680270da159 (127.3.75.2:42577)
I20260926 01:03:26.747165 21470 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.75.2@KRBTEST.COM'} at 127.3.75.2:34415
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab.
Entry for principal kudu/127.3.75.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab.
I20260926 01:03:26.761253 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.3:0
--local_ip_for_outbound_sockets=127.3.75.3
--webserver_interface=127.3.75.3
--webserver_port=0
--tserver_master_addrs=127.3.75.62:41399
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab
--principal=kudu/127.3.75.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:36235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:03:26.840826 21803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:03:26.841013 21803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:03:26.841035 21803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:03:26.842062 21803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:03:26.842094 21803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:03:26.842119 21803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.3
I20260926 01:03:26.843461 21803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:36235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab
--principal=kudu/127.3.75.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.75.3
--webserver_port=0
--tserver_master_addrs=127.3.75.62:41399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.3
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:03:26.843623 21803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:03:26.843779 21803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:03:26.848562 21811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:26.848716 21809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:03:26.848778 21803 server_base.cc:1034] running on GCE node
W20260926 01:03:26.848776 21808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:03:26.849040 21803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:03:26.849259 21803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:03:26.850420 21803 hybrid_clock.cc:648] HybridClock initialized: now 1790384606850414 us; error 34 us; skew 500 ppm
Sep 26 01:03:26 dist-test-slave-7v10 krb5kdc[21424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384606, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:03:26.853044 21803 init.cc:377] Logged in from keytab as kudu/127.3.75.3@KRBTEST.COM (short username kudu)
I20260926 01:03:26.853452 21803 webserver.cc:460] Webserver started at http://127.3.75.3:39005/ using document root <none> and password file <none>
I20260926 01:03:26.853672 21803 fs_manager.cc:359] Metadata directory not provided
I20260926 01:03:26.853714 21803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:03:26.853873 21803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:03:26.854961 21803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/ts-2/data/instance:
uuid: "93b907798b5040f2a462331c1f5251fa"
format_stamp: "Formatted at 2026-09-26 01:03:26 on dist-test-slave-7v10"
server_key: "7b99c28ce09f9a1505b1d69ec430baa2"
server_key_iv: "38600e65de6db76b394594e51647cf9e"
server_key_version: "encryptionkey@0"
I20260926 01:03:26.855332 21803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/ts-2/wal/instance:
uuid: "93b907798b5040f2a462331c1f5251fa"
format_stamp: "Formatted at 2026-09-26 01:03:26 on dist-test-slave-7v10"
server_key: "7b99c28ce09f9a1505b1d69ec430baa2"
server_key_iv: "38600e65de6db76b394594e51647cf9e"
server_key_version: "encryptionkey@0"
I20260926 01:03:26.856639 21803 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:03:26.857385 21818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:26.857601 21803 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:03:26.857700 21803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/ts-2/wal
uuid: "93b907798b5040f2a462331c1f5251fa"
format_stamp: "Formatted at 2026-09-26 01:03:26 on dist-test-slave-7v10"
server_key: "7b99c28ce09f9a1505b1d69ec430baa2"
server_key_iv: "38600e65de6db76b394594e51647cf9e"
server_key_version: "encryptionkey@0"
I20260926 01:03:26.857772 21803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:03:26.877667 21803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:03:26.879549 21803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:03:26.879698 21803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:03:26.879922 21803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:03:26.880225 21803 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:03:26.880275 21803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:26.880316 21803 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:03:26.880342 21803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:26.886178 21803 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.3:41983
I20260926 01:03:26.886224 21932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.3:41983 every 8 connection(s)
I20260926 01:03:26.886502 21803 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
Sep 26 01:03:26 dist-test-slave-7v10 krb5kdc[21424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384606, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.3@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 01:03:26.895041 21509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:26.887135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.3:49121 (local address 127.3.75.62:41399)
0926 01:03:26.887176 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:26.887178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:26.887364 (+ 186us) server_negotiation.cc:407] Connection header received
0926 01:03:26.887890 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:26.887891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:26.887905 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:26.887952 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:26.888878 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:26.889403 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:26.890370 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:26.890576 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:26.890664 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:26.892606 (+ 1942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:26.892608 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:26.892608 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:03:26.892615 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:26.893746 (+ 1131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:26.894116 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:26.894117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:26.894117 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:26.894179 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:26.894402 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:26.894403 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:26.894403 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:26.894512 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:26.894583 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:26.894832 (+ 249us) server_negotiation.cc:299] Negotiation successful
0926 01:03:26.894935 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 01:03:26.895210 21935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:26.887108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:41399 (local address 127.3.75.3:49121)
0926 01:03:26.887232 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:26.887286 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:26.887781 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:26.888031 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:26.888035 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:26.888593 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:03:26.888779 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:26.888784 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:26.889490 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:26.889491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:26.890274 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:26.890276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:26.890666 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:26.890667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:26.890785 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:26.891244 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:03:26.891252 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:03:26.892514 (+ 1262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:03:26.893822 (+ 1308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:03:26.893829 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:03:26.893832 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:03:26.894034 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:03:26.894242 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:03:26.894243 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:03:26.894243 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:03:26.894338 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:03:26.894573 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:03:26.894579 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:03:26.894708 (+ 129us) client_negotiation.cc:769] Sending connection context
0926 01:03:26.894801 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 01:03:26.894983 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":56,"threads_started":1}
I20260926 01:03:26.895645 21933 heartbeater.cc:347] Connected to a master server at 127.3.75.62:41399
I20260926 01:03:26.895740 21933 heartbeater.cc:464] Registering TS with master...
I20260926 01:03:26.895780 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 21803
I20260926 01:03:26.895908 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/ts-2/wal/instance
I20260926 01:03:26.895911 21933 heartbeater.cc:511] Master 127.3.75.62:41399 requested a full tablet report, sending...
I20260926 01:03:26.896114 3372 external_mini_cluster.cc:1428] Setting key 51b3e8a6cab5b03f2f9bfcb4ee1a9088
I20260926 01:03:26.896296 21470 ts_manager.cc:194] Registered new tserver with Master: 93b907798b5040f2a462331c1f5251fa (127.3.75.3:41983)
I20260926 01:03:26.896831 21470 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.75.3@KRBTEST.COM'} at 127.3.75.3:49121
I20260926 01:03:26.897498 3372 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 01:03:26.904819 21509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:26.898920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44402 (local address 127.3.75.62:41399)
0926 01:03:26.898958 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:26.898959 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:26.899082 (+ 123us) server_negotiation.cc:407] Connection header received
0926 01:03:26.899156 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:26.899157 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:26.899167 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:26.899221 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:26.899752 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:26.900202 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:26.900927 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:26.901123 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:26.901204 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:26.902368 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:26.902369 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:26.902370 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:03:26.902376 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:26.903670 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:26.904024 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:26.904025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:26.904026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:26.904080 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:26.904289 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:26.904290 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:26.904290 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:26.904412 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:26.904514 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:26.904632 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 01:03:26.904721 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 01:03:26.906360 21469 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44402:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:03:26.907917 21469 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:03:26.913594 21941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:26.909853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.2:42577 (local address 127.0.0.1:41662)
0926 01:03:26.909988 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:26.910023 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:26.910082 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:26.910270 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:26.910273 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:26.910276 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:03:26.910349 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:26.910351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:26.911490 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:26.911490 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:26.912429 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:26.912431 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:26.913295 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:26.913296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:26.913370 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:26.913382 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 01:03:26.913475 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 01:03:26.913526 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":86,"threads_started":1}
I20260926 01:03:26.913606 21943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:26.909962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41662 (local address 127.3.75.2:42577)
0926 01:03:26.910103 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:26.910107 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:26.910120 (+ 13us) server_negotiation.cc:407] Connection header received
0926 01:03:26.910163 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:26.910165 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:26.910180 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:26.910257 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:03:26.910446 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:26.911402 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:26.912516 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:26.913228 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:26.913294 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:26.913349 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:26.913492 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 01:03:26.913530 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":74,"threads_started":1}
I20260926 01:03:26.914122 21942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:26.909903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.3:41983 (local address 127.0.0.1:50082)
0926 01:03:26.910118 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:26.910129 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:26.910188 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:26.910585 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:26.910586 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:26.910588 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:03:26.910640 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:26.910642 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:26.912055 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:26.912056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:26.913006 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:26.913008 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:26.913857 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:26.913859 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:26.913937 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:26.913941 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:03:26.914018 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 01:03:26.914058 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":49,"threads_started":1}
I20260926 01:03:26.914162 21945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:26.910277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50082 (local address 127.3.75.3:41983)
0926 01:03:26.910450 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:26.910454 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:26.910463 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:03:26.910493 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:26.910495 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:26.910510 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:26.910587 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:03:26.910815 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:26.911952 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:26.913082 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:26.913775 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:26.913856 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:26.913911 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:26.914029 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 01:03:26.914066 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":90,"threads_started":1}
I20260926 01:03:26.914281 21944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:26.910062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.1:42299 (local address 127.0.0.1:53342)
0926 01:03:26.910338 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:26.910346 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:26.910400 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:26.910881 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:26.910882 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:26.910884 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:03:26.910965 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:26.910966 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:26.912153 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:26.912154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:26.913122 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:26.913124 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:26.914039 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:26.914040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:26.914109 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:26.914113 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:03:26.914164 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 01:03:26.914208 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":36,"threads_started":1}
I20260926 01:03:26.914507 21946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:26.910471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53342 (local address 127.3.75.1:42299)
0926 01:03:26.910661 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:26.910664 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:26.910674 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:03:26.910709 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:26.910711 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:26.910724 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:26.910795 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:03:26.911031 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:26.912069 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:26.913210 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:26.913872 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:26.913936 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:26.914003 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:26.914378 (+ 375us) server_negotiation.cc:299] Negotiation successful
0926 01:03:26.914425 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":77,"threads_started":1}
I20260926 01:03:26.914544 21732 tablet_service.cc:1467] Processing CreateTablet for tablet 9efd1a0580714b17b32af5a36ad0fa2a (DEFAULT_TABLE table=test-table [id=e5e235986cd54908a404c7420ddbe557]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:03:26.914963 21732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9efd1a0580714b17b32af5a36ad0fa2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:03:26.914870 21866 tablet_service.cc:1467] Processing CreateTablet for tablet 9efd1a0580714b17b32af5a36ad0fa2a (DEFAULT_TABLE table=test-table [id=e5e235986cd54908a404c7420ddbe557]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:03:26.915242 21866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9efd1a0580714b17b32af5a36ad0fa2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:03:26.917460 21947 tablet_bootstrap.cc:492] T 9efd1a0580714b17b32af5a36ad0fa2a P be9b12ad8128457bb6267680270da159: Bootstrap starting.
I20260926 01:03:26.917517 21948 tablet_bootstrap.cc:492] T 9efd1a0580714b17b32af5a36ad0fa2a P 93b907798b5040f2a462331c1f5251fa: Bootstrap starting.
I20260926 01:03:26.918197 21948 tablet_bootstrap.cc:654] T 9efd1a0580714b17b32af5a36ad0fa2a P 93b907798b5040f2a462331c1f5251fa: Neither blocks nor log segments found. Creating new log.
I20260926 01:03:26.918222 21947 tablet_bootstrap.cc:654] T 9efd1a0580714b17b32af5a36ad0fa2a P be9b12ad8128457bb6267680270da159: Neither blocks nor log segments found. Creating new log.
I20260926 01:03:26.918238 21597 tablet_service.cc:1467] Processing CreateTablet for tablet 9efd1a0580714b17b32af5a36ad0fa2a (DEFAULT_TABLE table=test-table [id=e5e235986cd54908a404c7420ddbe557]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:03:26.918584 21597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9efd1a0580714b17b32af5a36ad0fa2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:03:26.918769 21948 log.cc:824] T 9efd1a0580714b17b32af5a36ad0fa2a P 93b907798b5040f2a462331c1f5251fa: Log is configured to *not* fsync() on all Append() calls
I20260926 01:03:26.918793 21947 log.cc:824] T 9efd1a0580714b17b32af5a36ad0fa2a P be9b12ad8128457bb6267680270da159: Log is configured to *not* fsync() on all Append() calls
I20260926 01:03:26.919468 21947 tablet_bootstrap.cc:492] T 9efd1a0580714b17b32af5a36ad0fa2a P be9b12ad8128457bb6267680270da159: No bootstrap required, opened a new log
I20260926 01:03:26.919548 21947 ts_tablet_manager.cc:1397] T 9efd1a0580714b17b32af5a36ad0fa2a P be9b12ad8128457bb6267680270da159: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 01:03:26.919557 21948 tablet_bootstrap.cc:492] T 9efd1a0580714b17b32af5a36ad0fa2a P 93b907798b5040f2a462331c1f5251fa: No bootstrap required, opened a new log
I20260926 01:03:26.919613 21948 ts_tablet_manager.cc:1397] T 9efd1a0580714b17b32af5a36ad0fa2a P 93b907798b5040f2a462331c1f5251fa: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 01:03:26.920946 21947 raft_consensus.cc:348] T 9efd1a0580714b17b32af5a36ad0fa2a P be9b12ad8128457bb6267680270da159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be9b12ad8128457bb6267680270da159" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 42577 } } peers { permanent_uuid: "93b907798b5040f2a462331c1f5251fa" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41983 } } peers { permanent_uuid: "5f93e7d57c6a4c41a0d9bc91196e7887" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 42299 } }
I20260926 01:03:26.921108 21947 raft_consensus.cc:374] T 9efd1a0580714b17b32af5a36ad0fa2a P be9b12ad8128457bb6267680270da159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:03:26.921160 21947 raft_consensus.cc:729] T 9efd1a0580714b17b32af5a36ad0fa2a P be9b12ad8128457bb6267680270da159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be9b12ad8128457bb6267680270da159, State: Initialized, Role: FOLLOWER
I20260926 01:03:26.921288 21947 consensus_queue.cc:260] T 9efd1a0580714b17b32af5a36ad0fa2a P be9b12ad8128457bb6267680270da159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be9b12ad8128457bb6267680270da159" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 42577 } } peers { permanent_uuid: "93b907798b5040f2a462331c1f5251fa" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41983 } } peers { permanent_uuid: "5f93e7d57c6a4c41a0d9bc91196e7887" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 42299 } }
I20260926 01:03:26.921569 21947 ts_tablet_manager.cc:1428] T 9efd1a0580714b17b32af5a36ad0fa2a P be9b12ad8128457bb6267680270da159: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 01:03:26.921552 21948 raft_consensus.cc:348] T 9efd1a0580714b17b32af5a36ad0fa2a P 93b907798b5040f2a462331c1f5251fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be9b12ad8128457bb6267680270da159" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 42577 } } peers { permanent_uuid: "93b907798b5040f2a462331c1f5251fa" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41983 } } peers { permanent_uuid: "5f93e7d57c6a4c41a0d9bc91196e7887" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 42299 } }
I20260926 01:03:26.921702 21798 heartbeater.cc:503] Master 127.3.75.62:41399 was elected leader, sending a full tablet report...
I20260926 01:03:26.921726 21948 raft_consensus.cc:374] T 9efd1a0580714b17b32af5a36ad0fa2a P 93b907798b5040f2a462331c1f5251fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:03:26.921764 21948 raft_consensus.cc:729] T 9efd1a0580714b17b32af5a36ad0fa2a P 93b907798b5040f2a462331c1f5251fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93b907798b5040f2a462331c1f5251fa, State: Initialized, Role: FOLLOWER
I20260926 01:03:26.921840 21952 tablet_bootstrap.cc:492] T 9efd1a0580714b17b32af5a36ad0fa2a P 5f93e7d57c6a4c41a0d9bc91196e7887: Bootstrap starting.
I20260926 01:03:26.921875 21948 consensus_queue.cc:260] T 9efd1a0580714b17b32af5a36ad0fa2a P 93b907798b5040f2a462331c1f5251fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be9b12ad8128457bb6267680270da159" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 42577 } } peers { permanent_uuid: "93b907798b5040f2a462331c1f5251fa" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41983 } } peers { permanent_uuid: "5f93e7d57c6a4c41a0d9bc91196e7887" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 42299 } }
I20260926 01:03:26.922137 21948 ts_tablet_manager.cc:1428] T 9efd1a0580714b17b32af5a36ad0fa2a P 93b907798b5040f2a462331c1f5251fa: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 01:03:26.922281 21933 heartbeater.cc:503] Master 127.3.75.62:41399 was elected leader, sending a full tablet report...
I20260926 01:03:26.922495 21952 tablet_bootstrap.cc:654] T 9efd1a0580714b17b32af5a36ad0fa2a P 5f93e7d57c6a4c41a0d9bc91196e7887: Neither blocks nor log segments found. Creating new log.
I20260926 01:03:26.923076 21952 log.cc:824] T 9efd1a0580714b17b32af5a36ad0fa2a P 5f93e7d57c6a4c41a0d9bc91196e7887: Log is configured to *not* fsync() on all Append() calls
I20260926 01:03:26.923686 21952 tablet_bootstrap.cc:492] T 9efd1a0580714b17b32af5a36ad0fa2a P 5f93e7d57c6a4c41a0d9bc91196e7887: No bootstrap required, opened a new log
I20260926 01:03:26.923759 21952 ts_tablet_manager.cc:1397] T 9efd1a0580714b17b32af5a36ad0fa2a P 5f93e7d57c6a4c41a0d9bc91196e7887: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:03:26.925014 21952 raft_consensus.cc:348] T 9efd1a0580714b17b32af5a36ad0fa2a P 5f93e7d57c6a4c41a0d9bc91196e7887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be9b12ad8128457bb6267680270da159" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 42577 } } peers { permanent_uuid: "93b907798b5040f2a462331c1f5251fa" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41983 } } peers { permanent_uuid: "5f93e7d57c6a4c41a0d9bc91196e7887" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 42299 } }
I20260926 01:03:26.925168 21952 raft_consensus.cc:374] T 9efd1a0580714b17b32af5a36ad0fa2a P 5f93e7d57c6a4c41a0d9bc91196e7887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:03:26.925213 21952 raft_consensus.cc:729] T 9efd1a0580714b17b32af5a36ad0fa2a P 5f93e7d57c6a4c41a0d9bc91196e7887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f93e7d57c6a4c41a0d9bc91196e7887, State: Initialized, Role: FOLLOWER
I20260926 01:03:26.925405 21952 consensus_queue.cc:260] T 9efd1a0580714b17b32af5a36ad0fa2a P 5f93e7d57c6a4c41a0d9bc91196e7887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be9b12ad8128457bb6267680270da159" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 42577 } } peers { permanent_uuid: "93b907798b5040f2a462331c1f5251fa" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41983 } } peers { permanent_uuid: "5f93e7d57c6a4c41a0d9bc91196e7887" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 42299 } }
I20260926 01:03:26.925509 21955 raft_consensus.cc:482] T 9efd1a0580714b17b32af5a36ad0fa2a P 5f93e7d57c6a4c41a0d9bc91196e7887 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:03:26.925594 21955 raft_consensus.cc:504] T 9efd1a0580714b17b32af5a36ad0fa2a P 5f93e7d57c6a4c41a0d9bc91196e7887 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be9b12ad8128457bb6267680270da159" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 42577 } } peers { permanent_uuid: "93b907798b5040f2a462331c1f5251fa" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41983 } } peers { permanent_uuid: "5f93e7d57c6a4c41a0d9bc91196e7887" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 42299 } }
I20260926 01:03:26.925905 21952 ts_tablet_manager.cc:1428] T 9efd1a0580714b17b32af5a36ad0fa2a P 5f93e7d57c6a4c41a0d9bc91196e7887: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 01:03:26.925930 21663 heartbeater.cc:503] Master 127.3.75.62:41399 was elected leader, sending a full tablet report...
I20260926 01:03:26.926106 21955 leader_election.cc:288] T 9efd1a0580714b17b32af5a36ad0fa2a P 5f93e7d57c6a4c41a0d9bc91196e7887 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be9b12ad8128457bb6267680270da159 (127.3.75.2:42577), 93b907798b5040f2a462331c1f5251fa (127.3.75.3:41983)
I20260926 01:03:26.928910 21945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:26.926253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:38817 (local address 127.3.75.3:41983)
0926 01:03:26.926305 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:26.926307 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:26.926327 (+ 20us) server_negotiation.cc:407] Connection header received
0926 01:03:26.926347 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:26.926347 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:26.926357 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:26.926411 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:03:26.926673 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:26.927221 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:26.928182 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:26.928558 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:26.928643 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:26.928652 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:26.928816 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 01:03:26.928849 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 01:03:26.928910 21943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:26.926462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:58109 (local address 127.3.75.2:42577)
0926 01:03:26.926504 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:26.926506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:26.926513 (+ 7us) server_negotiation.cc:407] Connection header received
0926 01:03:26.926532 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:26.926533 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:26.926542 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:26.926590 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:03:26.926770 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:26.927298 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:26.928194 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:26.928565 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:26.928641 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:26.928657 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:26.928814 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 01:03:26.928847 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 01:03:26.928920 21665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:26.926090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.2:42577 (local address 127.3.75.1:58109)
0926 01:03:26.926152 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:26.926161 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:26.926196 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:26.926612 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:26.926627 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:26.926629 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:03:26.926695 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:26.926697 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:26.927383 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:26.927384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:26.928092 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:26.928094 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:26.928639 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:26.928640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:26.928710 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:26.928713 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:03:26.928807 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 01:03:26.928852 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 01:03:26.928937 21957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:26.926095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.3:41983 (local address 127.3.75.1:38817)
0926 01:03:26.926235 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:26.926245 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:26.926288 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:26.926412 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:26.926412 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:26.926413 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:03:26.926618 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:26.926620 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:26.927331 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:26.927332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:26.928098 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:26.928100 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:26.928646 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:26.928647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:26.928713 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:26.928717 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:03:26.928809 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 01:03:26.928847 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":52,"thread_start_us":73,"threads_started":1}
I20260926 01:03:26.929209 21752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9efd1a0580714b17b32af5a36ad0fa2a" candidate_uuid: "5f93e7d57c6a4c41a0d9bc91196e7887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be9b12ad8128457bb6267680270da159" is_pre_election: true
I20260926 01:03:26.929302 21752 raft_consensus.cc:2454] T 9efd1a0580714b17b32af5a36ad0fa2a P be9b12ad8128457bb6267680270da159 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f93e7d57c6a4c41a0d9bc91196e7887 in term 0.
I20260926 01:03:26.929318 21887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9efd1a0580714b17b32af5a36ad0fa2a" candidate_uuid: "5f93e7d57c6a4c41a0d9bc91196e7887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93b907798b5040f2a462331c1f5251fa" is_pre_election: true
I20260926 01:03:26.929445 21887 raft_consensus.cc:2454] T 9efd1a0580714b17b32af5a36ad0fa2a P 93b907798b5040f2a462331c1f5251fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f93e7d57c6a4c41a0d9bc91196e7887 in term 0.
I20260926 01:03:26.929486 21549 leader_election.cc:302] T 9efd1a0580714b17b32af5a36ad0fa2a P 5f93e7d57c6a4c41a0d9bc91196e7887 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f93e7d57c6a4c41a0d9bc91196e7887, be9b12ad8128457bb6267680270da159; no voters:
I20260926 01:03:26.929597 21955 raft_consensus.cc:2781] T 9efd1a0580714b17b32af5a36ad0fa2a P 5f93e7d57c6a4c41a0d9bc91196e7887 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:03:26.929634 21955 raft_consensus.cc:482] T 9efd1a0580714b17b32af5a36ad0fa2a P 5f93e7d57c6a4c41a0d9bc91196e7887 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:03:26.929652 21955 raft_consensus.cc:3037] T 9efd1a0580714b17b32af5a36ad0fa2a P 5f93e7d57c6a4c41a0d9bc91196e7887 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:03:26.930195 21955 raft_consensus.cc:504] T 9efd1a0580714b17b32af5a36ad0fa2a P 5f93e7d57c6a4c41a0d9bc91196e7887 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be9b12ad8128457bb6267680270da159" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 42577 } } peers { permanent_uuid: "93b907798b5040f2a462331c1f5251fa" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41983 } } peers { permanent_uuid: "5f93e7d57c6a4c41a0d9bc91196e7887" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 42299 } }
I20260926 01:03:26.930289 21955 leader_election.cc:288] T 9efd1a0580714b17b32af5a36ad0fa2a P 5f93e7d57c6a4c41a0d9bc91196e7887 [CANDIDATE]: Term 1 election: Requested vote from peers be9b12ad8128457bb6267680270da159 (127.3.75.2:42577), 93b907798b5040f2a462331c1f5251fa (127.3.75.3:41983)
I20260926 01:03:26.930433 21752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9efd1a0580714b17b32af5a36ad0fa2a" candidate_uuid: "5f93e7d57c6a4c41a0d9bc91196e7887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be9b12ad8128457bb6267680270da159"
I20260926 01:03:26.930470 21887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9efd1a0580714b17b32af5a36ad0fa2a" candidate_uuid: "5f93e7d57c6a4c41a0d9bc91196e7887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93b907798b5040f2a462331c1f5251fa"
I20260926 01:03:26.930505 21752 raft_consensus.cc:3037] T 9efd1a0580714b17b32af5a36ad0fa2a P be9b12ad8128457bb6267680270da159 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:03:26.930544 21887 raft_consensus.cc:3037] T 9efd1a0580714b17b32af5a36ad0fa2a P 93b907798b5040f2a462331c1f5251fa [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:03:26.931229 21887 raft_consensus.cc:2454] T 9efd1a0580714b17b32af5a36ad0fa2a P 93b907798b5040f2a462331c1f5251fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f93e7d57c6a4c41a0d9bc91196e7887 in term 1.
I20260926 01:03:26.931262 21752 raft_consensus.cc:2454] T 9efd1a0580714b17b32af5a36ad0fa2a P be9b12ad8128457bb6267680270da159 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f93e7d57c6a4c41a0d9bc91196e7887 in term 1.
I20260926 01:03:26.931442 21552 leader_election.cc:302] T 9efd1a0580714b17b32af5a36ad0fa2a P 5f93e7d57c6a4c41a0d9bc91196e7887 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f93e7d57c6a4c41a0d9bc91196e7887, 93b907798b5040f2a462331c1f5251fa; no voters:
I20260926 01:03:26.931536 21955 raft_consensus.cc:2781] T 9efd1a0580714b17b32af5a36ad0fa2a P 5f93e7d57c6a4c41a0d9bc91196e7887 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:03:26.931592 21955 raft_consensus.cc:686] T 9efd1a0580714b17b32af5a36ad0fa2a P 5f93e7d57c6a4c41a0d9bc91196e7887 [term 1 LEADER]: Becoming Leader. State: Replica: 5f93e7d57c6a4c41a0d9bc91196e7887, State: Running, Role: LEADER
I20260926 01:03:26.931669 21955 consensus_queue.cc:237] T 9efd1a0580714b17b32af5a36ad0fa2a P 5f93e7d57c6a4c41a0d9bc91196e7887 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be9b12ad8128457bb6267680270da159" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 42577 } } peers { permanent_uuid: "93b907798b5040f2a462331c1f5251fa" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41983 } } peers { permanent_uuid: "5f93e7d57c6a4c41a0d9bc91196e7887" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 42299 } }
I20260926 01:03:26.932399 21470 catalog_manager.cc:5495] T 9efd1a0580714b17b32af5a36ad0fa2a P 5f93e7d57c6a4c41a0d9bc91196e7887 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f93e7d57c6a4c41a0d9bc91196e7887 (127.3.75.1). New cstate: current_term: 1 leader_uuid: "5f93e7d57c6a4c41a0d9bc91196e7887" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be9b12ad8128457bb6267680270da159" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 42577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93b907798b5040f2a462331c1f5251fa" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f93e7d57c6a4c41a0d9bc91196e7887" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 42299 } health_report { overall_health: HEALTHY } } }
I20260926 01:03:26.942114 21946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:26.939841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53356 (local address 127.3.75.1:42299)
0926 01:03:26.939903 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:26.939905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:26.939914 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:03:26.939943 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:26.939943 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:26.939960 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:26.940037 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:03:26.940176 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:26.940635 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:26.941346 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:26.941549 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:26.941619 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:26.941792 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:03:26.941883 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:03:26.941974 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:26.942033 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 01:03:26.942041 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 01:03:26.943761 21887 raft_consensus.cc:1261] T 9efd1a0580714b17b32af5a36ad0fa2a P 93b907798b5040f2a462331c1f5251fa [term 1 FOLLOWER]: Refusing update from remote peer 5f93e7d57c6a4c41a0d9bc91196e7887: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:03:26.943761 21752 raft_consensus.cc:1261] T 9efd1a0580714b17b32af5a36ad0fa2a P be9b12ad8128457bb6267680270da159 [term 1 FOLLOWER]: Refusing update from remote peer 5f93e7d57c6a4c41a0d9bc91196e7887: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:03:26.944060 21956 consensus_queue.cc:1035] T 9efd1a0580714b17b32af5a36ad0fa2a P 5f93e7d57c6a4c41a0d9bc91196e7887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be9b12ad8128457bb6267680270da159" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 42577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:03:26.944199 21955 consensus_queue.cc:1035] T 9efd1a0580714b17b32af5a36ad0fa2a P 5f93e7d57c6a4c41a0d9bc91196e7887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93b907798b5040f2a462331c1f5251fa" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 41983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:03:26.948683 21469 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44402:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:03:26.948786 21469 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44402:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:03:26.949600 21469 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c54c1730e2c644cba915b8f409d4c659: Sending DeleteTablet for 3 replicas of tablet 9efd1a0580714b17b32af5a36ad0fa2a
I20260926 01:03:26.949944 21866 tablet_service.cc:1514] Processing DeleteTablet for tablet 9efd1a0580714b17b32af5a36ad0fa2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:03:26 UTC) from {username='kudu', principal='kudu/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:50082
I20260926 01:03:26.949990 21597 tablet_service.cc:1514] Processing DeleteTablet for tablet 9efd1a0580714b17b32af5a36ad0fa2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:03:26 UTC) from {username='kudu', principal='kudu/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:53342
I20260926 01:03:26.950234 21732 tablet_service.cc:1514] Processing DeleteTablet for tablet 9efd1a0580714b17b32af5a36ad0fa2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:03:26 UTC) from {username='kudu', principal='kudu/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:41662
I20260926 01:03:26.950510 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 21533
I20260926 01:03:26.950456 21970 tablet_replica.cc:330] T 9efd1a0580714b17b32af5a36ad0fa2a P 93b907798b5040f2a462331c1f5251fa: stopping tablet replica
I20260926 01:03:26.951202 21970 raft_consensus.cc:2229] T 9efd1a0580714b17b32af5a36ad0fa2a P 93b907798b5040f2a462331c1f5251fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:03:26.951305 21970 raft_consensus.cc:2258] T 9efd1a0580714b17b32af5a36ad0fa2a P 93b907798b5040f2a462331c1f5251fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:03:26.950616 21971 tablet_replica.cc:330] T 9efd1a0580714b17b32af5a36ad0fa2a P be9b12ad8128457bb6267680270da159: stopping tablet replica
I20260926 01:03:26.951552 21971 raft_consensus.cc:2229] T 9efd1a0580714b17b32af5a36ad0fa2a P be9b12ad8128457bb6267680270da159 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:03:26.951676 21971 raft_consensus.cc:2258] T 9efd1a0580714b17b32af5a36ad0fa2a P be9b12ad8128457bb6267680270da159 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:03:26.952087 21970 ts_tablet_manager.cc:1905] T 9efd1a0580714b17b32af5a36ad0fa2a P 93b907798b5040f2a462331c1f5251fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:03:26.953385 21971 ts_tablet_manager.cc:1905] T 9efd1a0580714b17b32af5a36ad0fa2a P be9b12ad8128457bb6267680270da159: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:03:26.953694 21970 ts_tablet_manager.cc:1918] T 9efd1a0580714b17b32af5a36ad0fa2a P 93b907798b5040f2a462331c1f5251fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:03:26.953771 21970 log.cc:1196] T 9efd1a0580714b17b32af5a36ad0fa2a P 93b907798b5040f2a462331c1f5251fa: Deleting WAL directory at /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/ts-2/wal/wals/9efd1a0580714b17b32af5a36ad0fa2a
I20260926 01:03:26.954057 21970 ts_tablet_manager.cc:1939] T 9efd1a0580714b17b32af5a36ad0fa2a P 93b907798b5040f2a462331c1f5251fa: Deleting consensus metadata
I20260926 01:03:26.954494 21457 catalog_manager.cc:4841] TS 93b907798b5040f2a462331c1f5251fa (127.3.75.3:41983): tablet 9efd1a0580714b17b32af5a36ad0fa2a (table test-table [id=e5e235986cd54908a404c7420ddbe557]) successfully deleted
I20260926 01:03:26.954972 21971 ts_tablet_manager.cc:1918] T 9efd1a0580714b17b32af5a36ad0fa2a P be9b12ad8128457bb6267680270da159: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:03:26.955065 21971 log.cc:1196] T 9efd1a0580714b17b32af5a36ad0fa2a P be9b12ad8128457bb6267680270da159: Deleting WAL directory at /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384372073861-3372-0/minicluster-data/ts-1/wal/wals/9efd1a0580714b17b32af5a36ad0fa2a
I20260926 01:03:26.955327 21971 ts_tablet_manager.cc:1939] T 9efd1a0580714b17b32af5a36ad0fa2a P be9b12ad8128457bb6267680270da159: Deleting consensus metadata
I20260926 01:03:26.955734 21454 catalog_manager.cc:4841] TS be9b12ad8128457bb6267680270da159 (127.3.75.2:42577): tablet 9efd1a0580714b17b32af5a36ad0fa2a (table test-table [id=e5e235986cd54908a404c7420ddbe557]) successfully deleted
W20260926 01:03:26.957515 21457 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.75.1:42299 (error 108)
I20260926 01:03:26.957720 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 21668
I20260926 01:03:26.958335 21944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:26.957823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.1:42299 (local address 127.0.0.1:53370)
0926 01:03:26.958262 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:26.958300 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.1:42299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416}
W20260926 01:03:26.958702 21457 catalog_manager.cc:4584] TS 5f93e7d57c6a4c41a0d9bc91196e7887 (127.3.75.1:42299): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9efd1a0580714b17b32af5a36ad0fa2a: Network error: Client connection negotiation failed: client connection to 127.3.75.1:42299: connect: Connection refused (error 111)
I20260926 01:03:26.963423 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 21803
I20260926 01:03:26.968029 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 21437
2026-09-26T01:03:26Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1848 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-26T01:03:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:03:26Z Disabled control of system clock
I20260926 01:03:26.994904 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:34535
--webserver_interface=127.3.75.62
--webserver_port=0
--builtin_ntp_servers=127.3.75.20:35993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.75.62:34535
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 01:03:27.075213 21976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:03:27.075417 21976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:03:27.075449 21976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:03:27.076460 21976 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:03:27.076491 21976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:03:27.076502 21976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:03:27.076514 21976 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:03:27.076531 21976 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:03:27.077759 21976 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:35993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.75.62:34535
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:34535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:03:27.077947 21976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:03:27.078171 21976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:03:27.083029 21981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:27.083041 21982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:27.083056 21984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:03:27.083267 21976 server_base.cc:1034] running on GCE node
I20260926 01:03:27.083572 21976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:03:27.083849 21976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:03:27.085022 21976 hybrid_clock.cc:648] HybridClock initialized: now 1790384607085009 us; error 32 us; skew 500 ppm
I20260926 01:03:27.086180 21976 webserver.cc:460] Webserver started at http://127.3.75.62:44077/ using document root <none> and password file <none>
I20260926 01:03:27.086431 21976 fs_manager.cc:359] Metadata directory not provided
I20260926 01:03:27.086498 21976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:03:27.086663 21976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:03:27.087764 21976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/master-0/data/instance:
uuid: "5eed5d13ee434329b69e8654b0777de9"
format_stamp: "Formatted at 2026-09-26 01:03:27 on dist-test-slave-7v10"
server_key: "b51ffd541581ff85bdccd6bc92cf201a"
server_key_iv: "74c56f1cebbd7f7a7c660b82e7250c94"
server_key_version: "encryptionkey@0"
I20260926 01:03:27.088121 21976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/master-0/wal/instance:
uuid: "5eed5d13ee434329b69e8654b0777de9"
format_stamp: "Formatted at 2026-09-26 01:03:27 on dist-test-slave-7v10"
server_key: "b51ffd541581ff85bdccd6bc92cf201a"
server_key_iv: "74c56f1cebbd7f7a7c660b82e7250c94"
server_key_version: "encryptionkey@0"
I20260926 01:03:27.089430 21976 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 01:03:27.090065 21990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:27.090222 21976 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 01:03:27.090312 21976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/master-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/master-0/wal
uuid: "5eed5d13ee434329b69e8654b0777de9"
format_stamp: "Formatted at 2026-09-26 01:03:27 on dist-test-slave-7v10"
server_key: "b51ffd541581ff85bdccd6bc92cf201a"
server_key_iv: "74c56f1cebbd7f7a7c660b82e7250c94"
server_key_version: "encryptionkey@0"
I20260926 01:03:27.090385 21976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:03:27.100706 21976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:03:27.101001 21976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:03:27.101147 21976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:03:27.105140 21976 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.62:34535
I20260926 01:03:27.105171 22043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.62:34535 every 8 connection(s)
I20260926 01:03:27.105506 21976 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
I20260926 01:03:27.106427 22044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:03:27.108690 22044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5eed5d13ee434329b69e8654b0777de9: Bootstrap starting.
I20260926 01:03:27.108952 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 21976
I20260926 01:03:27.109071 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/master-0/wal/instance
I20260926 01:03:27.109272 3372 external_mini_cluster.cc:1428] Setting key 9f35d77e3fabd5af97e6fc96b8e50a30
I20260926 01:03:27.109352 22044 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5eed5d13ee434329b69e8654b0777de9: Neither blocks nor log segments found. Creating new log.
I20260926 01:03:27.110090 22044 log.cc:824] T 00000000000000000000000000000000 P 5eed5d13ee434329b69e8654b0777de9: Log is configured to *not* fsync() on all Append() calls
I20260926 01:03:27.110852 22044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5eed5d13ee434329b69e8654b0777de9: No bootstrap required, opened a new log
I20260926 01:03:27.112568 22044 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5eed5d13ee434329b69e8654b0777de9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eed5d13ee434329b69e8654b0777de9" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 34535 } }
I20260926 01:03:27.112721 22044 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5eed5d13ee434329b69e8654b0777de9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:03:27.112769 22044 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5eed5d13ee434329b69e8654b0777de9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5eed5d13ee434329b69e8654b0777de9, State: Initialized, Role: FOLLOWER
I20260926 01:03:27.112888 22044 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5eed5d13ee434329b69e8654b0777de9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eed5d13ee434329b69e8654b0777de9" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 34535 } }
I20260926 01:03:27.112977 22044 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5eed5d13ee434329b69e8654b0777de9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:03:27.113019 22044 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5eed5d13ee434329b69e8654b0777de9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:03:27.113065 22044 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5eed5d13ee434329b69e8654b0777de9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:03:27.113545 22044 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5eed5d13ee434329b69e8654b0777de9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eed5d13ee434329b69e8654b0777de9" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 34535 } }
I20260926 01:03:27.113653 22044 leader_election.cc:302] T 00000000000000000000000000000000 P 5eed5d13ee434329b69e8654b0777de9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5eed5d13ee434329b69e8654b0777de9; no voters:
I20260926 01:03:27.113837 22044 leader_election.cc:288] T 00000000000000000000000000000000 P 5eed5d13ee434329b69e8654b0777de9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:03:27.113875 22049 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5eed5d13ee434329b69e8654b0777de9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:03:27.114010 22049 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5eed5d13ee434329b69e8654b0777de9 [term 1 LEADER]: Becoming Leader. State: Replica: 5eed5d13ee434329b69e8654b0777de9, State: Running, Role: LEADER
I20260926 01:03:27.114095 22044 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5eed5d13ee434329b69e8654b0777de9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:03:27.114101 22049 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5eed5d13ee434329b69e8654b0777de9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eed5d13ee434329b69e8654b0777de9" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 34535 } }
I20260926 01:03:27.114466 22050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5eed5d13ee434329b69e8654b0777de9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5eed5d13ee434329b69e8654b0777de9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eed5d13ee434329b69e8654b0777de9" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 34535 } } }
I20260926 01:03:27.114466 22051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5eed5d13ee434329b69e8654b0777de9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5eed5d13ee434329b69e8654b0777de9. Latest consensus state: current_term: 1 leader_uuid: "5eed5d13ee434329b69e8654b0777de9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eed5d13ee434329b69e8654b0777de9" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 34535 } } }
I20260926 01:03:27.114573 22050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5eed5d13ee434329b69e8654b0777de9 [sys.catalog]: This master's current role is: LEADER
I20260926 01:03:27.114665 22051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5eed5d13ee434329b69e8654b0777de9 [sys.catalog]: This master's current role is: LEADER
I20260926 01:03:27.115054 22047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:27.109748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40016 (local address 127.3.75.62:34535)
0926 01:03:27.109987 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:27.109996 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:27.110031 (+ 35us) server_negotiation.cc:407] Connection header received
0926 01:03:27.110544 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:27.110553 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:27.110630 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:27.110780 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:27.111033 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:27.111937 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:27.112699 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:27.113249 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:27.113370 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:27.114100 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:27.114104 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:27.114105 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:03:27.114324 (+ 219us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:27.114370 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:03:27.114384 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:27.114442 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:27.114652 (+ 210us) server_negotiation.cc:299] Negotiation successful
0926 01:03:27.114737 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":92,"threads_started":1}
I20260926 01:03:27.115378 22054 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:03:27.116005 22054 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:03:27.117398 22054 catalog_manager.cc:1347] Generated new cluster ID: 4e48640ce16142bea4ba0d1c75606aea
I20260926 01:03:27.117453 22054 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:03:27.123803 22054 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:03:27.124305 22054 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:03:27.134431 22054 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5eed5d13ee434329b69e8654b0777de9: Generated new TSK 0
I20260926 01:03:27.134595 22054 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:03:27.140074 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.1:0
--local_ip_for_outbound_sockets=127.3.75.1
--webserver_interface=127.3.75.1
--webserver_port=0
--tserver_master_addrs=127.3.75.62:34535
--builtin_ntp_servers=127.3.75.20:35993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 01:03:27.220178 22068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:03:27.220367 22068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:03:27.220387 22068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:03:27.221479 22068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:03:27.221517 22068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:03:27.221541 22068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.1
I20260926 01:03:27.222842 22068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:35993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.75.1
--webserver_port=0
--tserver_master_addrs=127.3.75.62:34535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.1
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:03:27.223151 22068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:03:27.223441 22068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:03:27.229296 22074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:27.229300 22073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:27.229460 22076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:03:27.229566 22068 server_base.cc:1034] running on GCE node
I20260926 01:03:27.229758 22068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:03:27.230006 22068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:03:27.231158 22068 hybrid_clock.cc:648] HybridClock initialized: now 1790384607231137 us; error 37 us; skew 500 ppm
I20260926 01:03:27.232378 22068 webserver.cc:460] Webserver started at http://127.3.75.1:41487/ using document root <none> and password file <none>
I20260926 01:03:27.232640 22068 fs_manager.cc:359] Metadata directory not provided
I20260926 01:03:27.232709 22068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:03:27.232831 22068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:03:27.233832 22068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/ts-0/data/instance:
uuid: "a8e18660a84c4a89881e04dd110ba577"
format_stamp: "Formatted at 2026-09-26 01:03:27 on dist-test-slave-7v10"
server_key: "750383c406c686b746ee35ee8ae1177f"
server_key_iv: "9ad6d6c017239753b4ec59b9aae55891"
server_key_version: "encryptionkey@0"
I20260926 01:03:27.234166 22068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/ts-0/wal/instance:
uuid: "a8e18660a84c4a89881e04dd110ba577"
format_stamp: "Formatted at 2026-09-26 01:03:27 on dist-test-slave-7v10"
server_key: "750383c406c686b746ee35ee8ae1177f"
server_key_iv: "9ad6d6c017239753b4ec59b9aae55891"
server_key_version: "encryptionkey@0"
I20260926 01:03:27.235548 22068 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:03:27.236332 22082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:27.236518 22068 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:03:27.236598 22068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/ts-0/wal
uuid: "a8e18660a84c4a89881e04dd110ba577"
format_stamp: "Formatted at 2026-09-26 01:03:27 on dist-test-slave-7v10"
server_key: "750383c406c686b746ee35ee8ae1177f"
server_key_iv: "9ad6d6c017239753b4ec59b9aae55891"
server_key_version: "encryptionkey@0"
I20260926 01:03:27.236649 22068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:03:27.246279 22068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:03:27.246495 22068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:03:27.246594 22068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:03:27.246754 22068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:03:27.247071 22068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:03:27.247103 22068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:27.247124 22068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:03:27.247143 22068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:27.252414 22068 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.1:44585
I20260926 01:03:27.252502 22195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.1:44585 every 8 connection(s)
I20260926 01:03:27.252792 22068 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
I20260926 01:03:27.254669 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 22068
I20260926 01:03:27.254761 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/ts-0/wal/instance
I20260926 01:03:27.254997 3372 external_mini_cluster.cc:1428] Setting key 5f29a9ee2cecac9d6cc41fc4a0cb3d55
I20260926 01:03:27.255891 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.2:0
--local_ip_for_outbound_sockets=127.3.75.2
--webserver_interface=127.3.75.2
--webserver_port=0
--tserver_master_addrs=127.3.75.62:34535
--builtin_ntp_servers=127.3.75.20:35993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 01:03:27.259065 22047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:27.253476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:45961 (local address 127.3.75.62:34535)
0926 01:03:27.253533 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:27.253535 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:27.253759 (+ 224us) server_negotiation.cc:407] Connection header received
0926 01:03:27.254312 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:27.254314 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:27.254331 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:27.254391 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:27.255282 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:27.255984 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:27.257491 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:27.257820 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:27.257901 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:27.258692 (+ 791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:27.258693 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:27.258694 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:03:27.258699 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:27.258717 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:03:27.258721 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:27.258773 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:27.258858 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 01:03:27.258947 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 01:03:27.259171 22198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:27.253468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34535 (local address 127.3.75.1:45961)
0926 01:03:27.253652 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:27.253683 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:27.254203 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:27.254457 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:27.254461 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:27.254475 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:03:27.255153 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:27.255159 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:27.256091 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:27.256093 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:27.257408 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:27.257412 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:27.257907 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:27.257907 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:27.258016 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:27.258551 (+ 535us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 01:03:27.258551 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:03:27.258576 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 01:03:27.258598 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:03:27.258788 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:03:27.258792 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:03:27.258795 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:03:27.258843 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 01:03:27.258990 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":81,"threads_started":1}
I20260926 01:03:27.259660 22196 heartbeater.cc:347] Connected to a master server at 127.3.75.62:34535
I20260926 01:03:27.259756 22196 heartbeater.cc:464] Registering TS with master...
I20260926 01:03:27.259959 22196 heartbeater.cc:511] Master 127.3.75.62:34535 requested a full tablet report, sending...
I20260926 01:03:27.260528 22007 ts_manager.cc:194] Registered new tserver with Master: a8e18660a84c4a89881e04dd110ba577 (127.3.75.1:44585)
I20260926 01:03:27.261518 22007 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.75.1:45961
W20260926 01:03:27.338582 22200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:03:27.338771 22200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:03:27.338793 22200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:03:27.340109 22200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:03:27.340165 22200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:03:27.340194 22200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.2
I20260926 01:03:27.341547 22200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:35993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.75.2
--webserver_port=0
--tserver_master_addrs=127.3.75.62:34535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.2
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:03:27.341737 22200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:03:27.341976 22200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:03:27.346625 22208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:27.346700 22206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:27.346598 22205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:03:27.346721 22200 server_base.cc:1034] running on GCE node
I20260926 01:03:27.347078 22200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:03:27.347383 22200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:03:27.348532 22200 hybrid_clock.cc:648] HybridClock initialized: now 1790384607348519 us; error 34 us; skew 500 ppm
I20260926 01:03:27.349728 22200 webserver.cc:460] Webserver started at http://127.3.75.2:37503/ using document root <none> and password file <none>
I20260926 01:03:27.349959 22200 fs_manager.cc:359] Metadata directory not provided
I20260926 01:03:27.350019 22200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:03:27.350212 22200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:03:27.351282 22200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/ts-1/data/instance:
uuid: "9fb55caaa9474d899c980fdc9965eac6"
format_stamp: "Formatted at 2026-09-26 01:03:27 on dist-test-slave-7v10"
server_key: "57f8d03e147f3348bc245c9db5561d70"
server_key_iv: "653a56c92e4d6e80b292dd4b9024c08c"
server_key_version: "encryptionkey@0"
I20260926 01:03:27.351616 22200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/ts-1/wal/instance:
uuid: "9fb55caaa9474d899c980fdc9965eac6"
format_stamp: "Formatted at 2026-09-26 01:03:27 on dist-test-slave-7v10"
server_key: "57f8d03e147f3348bc245c9db5561d70"
server_key_iv: "653a56c92e4d6e80b292dd4b9024c08c"
server_key_version: "encryptionkey@0"
I20260926 01:03:27.352833 22200 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 01:03:27.353495 22214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:27.353650 22200 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 01:03:27.353745 22200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/ts-1/wal
uuid: "9fb55caaa9474d899c980fdc9965eac6"
format_stamp: "Formatted at 2026-09-26 01:03:27 on dist-test-slave-7v10"
server_key: "57f8d03e147f3348bc245c9db5561d70"
server_key_iv: "653a56c92e4d6e80b292dd4b9024c08c"
server_key_version: "encryptionkey@0"
I20260926 01:03:27.353817 22200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:03:27.383080 22200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:03:27.383380 22200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:03:27.383518 22200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:03:27.383749 22200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:03:27.384076 22200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:03:27.384125 22200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:27.384187 22200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:03:27.384218 22200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:27.389741 22200 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.2:44587
I20260926 01:03:27.389786 22329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.2:44587 every 8 connection(s)
I20260926 01:03:27.390103 22200 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
I20260926 01:03:27.391665 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 22200
I20260926 01:03:27.391784 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/ts-1/wal/instance
I20260926 01:03:27.391990 3372 external_mini_cluster.cc:1428] Setting key 7dd2fa143e551962960e76b79f7c375a
I20260926 01:03:27.392959 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.3:0
--local_ip_for_outbound_sockets=127.3.75.3
--webserver_interface=127.3.75.3
--webserver_port=0
--tserver_master_addrs=127.3.75.62:34535
--builtin_ntp_servers=127.3.75.20:35993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 01:03:27.396095 22047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:27.390860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:54193 (local address 127.3.75.62:34535)
0926 01:03:27.390938 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:27.390940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:27.391159 (+ 219us) server_negotiation.cc:407] Connection header received
0926 01:03:27.391783 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:27.391785 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:27.391798 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:27.391876 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:27.392721 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:27.393497 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:27.394719 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:27.394956 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:27.395012 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:27.395877 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:27.395878 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:27.395878 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:03:27.395882 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:27.395896 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:03:27.395899 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:27.395940 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:27.396019 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 01:03:27.396039 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 01:03:27.396386 22332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:27.390856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34535 (local address 127.3.75.2:54193)
0926 01:03:27.391041 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:27.391073 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:27.391636 (+ 563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:27.391987 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:27.391991 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:27.391998 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:03:27.392622 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:27.392629 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:27.393586 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:27.393587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:27.394628 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:27.394630 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:27.395032 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:27.395033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:27.395214 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:27.395732 (+ 518us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 01:03:27.395732 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:03:27.395770 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 01:03:27.395796 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:03:27.395941 (+ 145us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:03:27.395945 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:03:27.395947 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 01:03:27.396025 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 01:03:27.396130 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":106,"threads_started":1}
I20260926 01:03:27.396844 22330 heartbeater.cc:347] Connected to a master server at 127.3.75.62:34535
I20260926 01:03:27.396946 22330 heartbeater.cc:464] Registering TS with master...
I20260926 01:03:27.397121 22330 heartbeater.cc:511] Master 127.3.75.62:34535 requested a full tablet report, sending...
I20260926 01:03:27.397529 22007 ts_manager.cc:194] Registered new tserver with Master: 9fb55caaa9474d899c980fdc9965eac6 (127.3.75.2:44587)
I20260926 01:03:27.397889 22007 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.75.2:54193
W20260926 01:03:27.473790 22333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:03:27.473989 22333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:03:27.474010 22333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:03:27.475162 22333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:03:27.475199 22333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:03:27.475224 22333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.3
I20260926 01:03:27.476548 22333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:35993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.75.3
--webserver_port=0
--tserver_master_addrs=127.3.75.62:34535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.3
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:03:27.476823 22333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:03:27.477105 22333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:03:27.481851 22338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:27.481859 22341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:27.482002 22339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:03:27.482090 22333 server_base.cc:1034] running on GCE node
I20260926 01:03:27.482303 22333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:03:27.482525 22333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:03:27.483680 22333 hybrid_clock.cc:648] HybridClock initialized: now 1790384607483662 us; error 30 us; skew 500 ppm
I20260926 01:03:27.488514 22333 webserver.cc:460] Webserver started at http://127.3.75.3:37339/ using document root <none> and password file <none>
I20260926 01:03:27.488757 22333 fs_manager.cc:359] Metadata directory not provided
I20260926 01:03:27.488821 22333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:03:27.488958 22333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:03:27.489934 22333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/ts-2/data/instance:
uuid: "9289839d25724333854785220983c1bb"
format_stamp: "Formatted at 2026-09-26 01:03:27 on dist-test-slave-7v10"
server_key: "61fee46ced3bf4834d8514891ceccaf6"
server_key_iv: "d9e2978e2c295c36b69be1c7f4fba879"
server_key_version: "encryptionkey@0"
I20260926 01:03:27.490274 22333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/ts-2/wal/instance:
uuid: "9289839d25724333854785220983c1bb"
format_stamp: "Formatted at 2026-09-26 01:03:27 on dist-test-slave-7v10"
server_key: "61fee46ced3bf4834d8514891ceccaf6"
server_key_iv: "d9e2978e2c295c36b69be1c7f4fba879"
server_key_version: "encryptionkey@0"
I20260926 01:03:27.491539 22333 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 01:03:27.492276 22347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:27.492437 22333 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 01:03:27.492539 22333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/ts-2/wal
uuid: "9289839d25724333854785220983c1bb"
format_stamp: "Formatted at 2026-09-26 01:03:27 on dist-test-slave-7v10"
server_key: "61fee46ced3bf4834d8514891ceccaf6"
server_key_iv: "d9e2978e2c295c36b69be1c7f4fba879"
server_key_version: "encryptionkey@0"
I20260926 01:03:27.492614 22333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:03:27.501511 22333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:03:27.501753 22333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:03:27.501884 22333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:03:27.502072 22333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:03:27.502359 22333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:03:27.502408 22333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:27.502446 22333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:03:27.502473 22333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:27.507965 22333 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.3:44757
I20260926 01:03:27.508009 22461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.3:44757 every 8 connection(s)
I20260926 01:03:27.508306 22333 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
I20260926 01:03:27.513409 22047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:27.508958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.3:42651 (local address 127.3.75.62:34535)
0926 01:03:27.509002 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:27.509004 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:27.509211 (+ 207us) server_negotiation.cc:407] Connection header received
0926 01:03:27.509734 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:27.509736 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:27.509750 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:27.509809 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:27.510472 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:27.511026 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:27.512069 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:27.512285 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:27.512365 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:27.513161 (+ 796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:27.513163 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:27.513163 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:03:27.513168 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:27.513183 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:03:27.513186 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:27.513234 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:27.513306 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 01:03:27.513331 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 01:03:27.513607 22464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:27.508933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:34535 (local address 127.3.75.3:42651)
0926 01:03:27.509095 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:27.509125 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:27.509612 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:27.509876 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:27.509880 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:27.509889 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:03:27.510380 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:27.510384 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:27.511127 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:27.511128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:27.511972 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:27.511974 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:27.512376 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:27.512377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:27.512507 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:27.513018 (+ 511us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 01:03:27.513018 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:03:27.513045 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 01:03:27.513065 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:03:27.513234 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:03:27.513239 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:03:27.513242 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:03:27.513305 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 01:03:27.513401 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":71,"threads_started":1}
I20260926 01:03:27.514084 22462 heartbeater.cc:347] Connected to a master server at 127.3.75.62:34535
I20260926 01:03:27.514168 22462 heartbeater.cc:464] Registering TS with master...
I20260926 01:03:27.514309 22462 heartbeater.cc:511] Master 127.3.75.62:34535 requested a full tablet report, sending...
I20260926 01:03:27.514647 22007 ts_manager.cc:194] Registered new tserver with Master: 9289839d25724333854785220983c1bb (127.3.75.3:44757)
I20260926 01:03:27.515035 22007 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.75.3:42651
I20260926 01:03:27.518419 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 22333
I20260926 01:03:27.518504 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/ts-2/wal/instance
I20260926 01:03:27.518657 3372 external_mini_cluster.cc:1428] Setting key 4bd4ce46c711dea967af3ea336c6e0dc
I20260926 01:03:27.519470 3372 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 01:03:27.523528 22047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:27.520761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40018 (local address 127.3.75.62:34535)
0926 01:03:27.520833 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:27.520835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:27.520935 (+ 100us) server_negotiation.cc:407] Connection header received
0926 01:03:27.520999 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:27.521000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:27.521009 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:27.521062 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:27.521232 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:27.521651 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:27.522335 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:27.522519 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:27.522591 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:27.523275 (+ 684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:27.523276 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:27.523276 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:03:27.523280 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:27.523293 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:03:27.523296 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:27.523386 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:27.523450 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 01:03:27.523459 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 01:03:27.524935 22007 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40018:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:03:27.526290 22007 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:03:27.532153 22474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:27.528374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.2:44587 (local address 127.0.0.1:32892)
0926 01:03:27.528484 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:27.528494 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:27.528758 (+ 264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:27.528932 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:27.528933 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:27.528935 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:03:27.528979 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:27.528980 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:27.530245 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:27.530246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:27.531125 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:27.531127 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:27.531927 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:27.531927 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:27.531980 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:27.531991 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 01:03:27.532063 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 01:03:27.532102 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"mutex_wait_us":1,"thread_start_us":65,"threads_started":1}
I20260926 01:03:27.532202 22475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:27.528589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32892 (local address 127.3.75.2:44587)
0926 01:03:27.528748 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:27.528752 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:27.528759 (+ 7us) server_negotiation.cc:407] Connection header received
0926 01:03:27.528829 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:27.528831 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:27.528846 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:27.528925 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:03:27.529093 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:27.530189 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:27.531216 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:27.531838 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:27.531907 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:27.531984 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:27.532089 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 01:03:27.532124 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":105,"threads_started":1}
I20260926 01:03:27.532615 22471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:27.528202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.1:44585 (local address 127.0.0.1:39740)
0926 01:03:27.528391 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:27.528425 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:27.528490 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:27.528825 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:27.528827 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:27.528829 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:03:27.528900 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:27.528902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:27.530206 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:27.530207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:27.531478 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:27.531481 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:27.532398 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:27.532399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:27.532453 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:27.532457 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:03:27.532507 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 01:03:27.532543 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":32,"thread_start_us":45,"threads_started":1}
I20260926 01:03:27.532979 22473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:27.528288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39740 (local address 127.3.75.1:44585)
0926 01:03:27.528649 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:27.528655 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:27.528665 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:03:27.528704 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:27.528705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:27.528720 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:27.528804 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:03:27.528981 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:27.530128 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:27.531608 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:27.532342 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:27.532395 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:27.532445 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:27.532857 (+ 412us) server_negotiation.cc:299] Negotiation successful
0926 01:03:27.532901 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":87,"threads_started":1}
I20260926 01:03:27.533090 22263 tablet_service.cc:1467] Processing CreateTablet for tablet e946e56dc81e45a5ada9f1847889d2c2 (DEFAULT_TABLE table=test-table [id=39ac2420278348de8d3b4b6484eb5c6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:03:27.533237 22470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:27.528234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.3:44757 (local address 127.0.0.1:58280)
0926 01:03:27.528471 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:27.528483 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:27.528544 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:27.529238 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:27.529239 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:27.529241 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:03:27.529274 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:27.529275 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:27.530109 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:27.530110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:27.531478 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:27.531481 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:27.533011 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:27.533012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:27.533068 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:27.533071 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:03:27.533125 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 01:03:27.533162 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":42,"thread_start_us":87,"threads_started":1}
I20260926 01:03:27.533464 22263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e946e56dc81e45a5ada9f1847889d2c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:03:27.533758 22130 tablet_service.cc:1467] Processing CreateTablet for tablet e946e56dc81e45a5ada9f1847889d2c2 (DEFAULT_TABLE table=test-table [id=39ac2420278348de8d3b4b6484eb5c6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:03:27.533979 22130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e946e56dc81e45a5ada9f1847889d2c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:03:27.535964 22476 tablet_bootstrap.cc:492] T e946e56dc81e45a5ada9f1847889d2c2 P 9fb55caaa9474d899c980fdc9965eac6: Bootstrap starting.
I20260926 01:03:27.536150 22477 tablet_bootstrap.cc:492] T e946e56dc81e45a5ada9f1847889d2c2 P a8e18660a84c4a89881e04dd110ba577: Bootstrap starting.
I20260926 01:03:27.536410 22472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:27.528300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58280 (local address 127.3.75.3:44757)
0926 01:03:27.529053 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:27.529056 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:27.529065 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:03:27.529098 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:27.529099 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:27.529114 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:27.529365 (+ 251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:03:27.529397 (+ 32us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:27.530030 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:27.532470 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:27.532934 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:27.533010 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:27.533079 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:27.533137 (+ 58us) server_negotiation.cc:299] Negotiation successful
0926 01:03:27.536318 (+ 3181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":64,"threads_started":1}
I20260926 01:03:27.536689 22476 tablet_bootstrap.cc:654] T e946e56dc81e45a5ada9f1847889d2c2 P 9fb55caaa9474d899c980fdc9965eac6: Neither blocks nor log segments found. Creating new log.
I20260926 01:03:27.536787 22477 tablet_bootstrap.cc:654] T e946e56dc81e45a5ada9f1847889d2c2 P a8e18660a84c4a89881e04dd110ba577: Neither blocks nor log segments found. Creating new log.
I20260926 01:03:27.536970 22396 tablet_service.cc:1467] Processing CreateTablet for tablet e946e56dc81e45a5ada9f1847889d2c2 (DEFAULT_TABLE table=test-table [id=39ac2420278348de8d3b4b6484eb5c6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:03:27.537065 22476 log.cc:824] T e946e56dc81e45a5ada9f1847889d2c2 P 9fb55caaa9474d899c980fdc9965eac6: Log is configured to *not* fsync() on all Append() calls
I20260926 01:03:27.537094 22477 log.cc:824] T e946e56dc81e45a5ada9f1847889d2c2 P a8e18660a84c4a89881e04dd110ba577: Log is configured to *not* fsync() on all Append() calls
I20260926 01:03:27.537242 22396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e946e56dc81e45a5ada9f1847889d2c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:03:27.537798 22477 tablet_bootstrap.cc:492] T e946e56dc81e45a5ada9f1847889d2c2 P a8e18660a84c4a89881e04dd110ba577: No bootstrap required, opened a new log
I20260926 01:03:27.537858 22476 tablet_bootstrap.cc:492] T e946e56dc81e45a5ada9f1847889d2c2 P 9fb55caaa9474d899c980fdc9965eac6: No bootstrap required, opened a new log
I20260926 01:03:27.537884 22477 ts_tablet_manager.cc:1397] T e946e56dc81e45a5ada9f1847889d2c2 P a8e18660a84c4a89881e04dd110ba577: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:03:27.537921 22476 ts_tablet_manager.cc:1397] T e946e56dc81e45a5ada9f1847889d2c2 P 9fb55caaa9474d899c980fdc9965eac6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 01:03:27.539785 22477 raft_consensus.cc:348] T e946e56dc81e45a5ada9f1847889d2c2 P a8e18660a84c4a89881e04dd110ba577 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8e18660a84c4a89881e04dd110ba577" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 44585 } } peers { permanent_uuid: "9fb55caaa9474d899c980fdc9965eac6" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 44587 } } peers { permanent_uuid: "9289839d25724333854785220983c1bb" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 44757 } }
I20260926 01:03:27.539902 22477 raft_consensus.cc:374] T e946e56dc81e45a5ada9f1847889d2c2 P a8e18660a84c4a89881e04dd110ba577 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:03:27.539925 22477 raft_consensus.cc:729] T e946e56dc81e45a5ada9f1847889d2c2 P a8e18660a84c4a89881e04dd110ba577 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8e18660a84c4a89881e04dd110ba577, State: Initialized, Role: FOLLOWER
I20260926 01:03:27.540037 22477 consensus_queue.cc:260] T e946e56dc81e45a5ada9f1847889d2c2 P a8e18660a84c4a89881e04dd110ba577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8e18660a84c4a89881e04dd110ba577" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 44585 } } peers { permanent_uuid: "9fb55caaa9474d899c980fdc9965eac6" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 44587 } } peers { permanent_uuid: "9289839d25724333854785220983c1bb" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 44757 } }
I20260926 01:03:27.540283 22477 ts_tablet_manager.cc:1428] T e946e56dc81e45a5ada9f1847889d2c2 P a8e18660a84c4a89881e04dd110ba577: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260926 01:03:27.540304 22196 heartbeater.cc:503] Master 127.3.75.62:34535 was elected leader, sending a full tablet report...
I20260926 01:03:27.540520 22476 raft_consensus.cc:348] T e946e56dc81e45a5ada9f1847889d2c2 P 9fb55caaa9474d899c980fdc9965eac6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8e18660a84c4a89881e04dd110ba577" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 44585 } } peers { permanent_uuid: "9fb55caaa9474d899c980fdc9965eac6" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 44587 } } peers { permanent_uuid: "9289839d25724333854785220983c1bb" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 44757 } }
I20260926 01:03:27.540619 22481 tablet_bootstrap.cc:492] T e946e56dc81e45a5ada9f1847889d2c2 P 9289839d25724333854785220983c1bb: Bootstrap starting.
I20260926 01:03:27.540665 22476 raft_consensus.cc:374] T e946e56dc81e45a5ada9f1847889d2c2 P 9fb55caaa9474d899c980fdc9965eac6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:03:27.540688 22476 raft_consensus.cc:729] T e946e56dc81e45a5ada9f1847889d2c2 P 9fb55caaa9474d899c980fdc9965eac6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fb55caaa9474d899c980fdc9965eac6, State: Initialized, Role: FOLLOWER
I20260926 01:03:27.540788 22476 consensus_queue.cc:260] T e946e56dc81e45a5ada9f1847889d2c2 P 9fb55caaa9474d899c980fdc9965eac6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8e18660a84c4a89881e04dd110ba577" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 44585 } } peers { permanent_uuid: "9fb55caaa9474d899c980fdc9965eac6" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 44587 } } peers { permanent_uuid: "9289839d25724333854785220983c1bb" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 44757 } }
I20260926 01:03:27.540979 22476 ts_tablet_manager.cc:1428] T e946e56dc81e45a5ada9f1847889d2c2 P 9fb55caaa9474d899c980fdc9965eac6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 01:03:27.541074 22330 heartbeater.cc:503] Master 127.3.75.62:34535 was elected leader, sending a full tablet report...
I20260926 01:03:27.541152 22481 tablet_bootstrap.cc:654] T e946e56dc81e45a5ada9f1847889d2c2 P 9289839d25724333854785220983c1bb: Neither blocks nor log segments found. Creating new log.
I20260926 01:03:27.541479 22481 log.cc:824] T e946e56dc81e45a5ada9f1847889d2c2 P 9289839d25724333854785220983c1bb: Log is configured to *not* fsync() on all Append() calls
I20260926 01:03:27.542306 22481 tablet_bootstrap.cc:492] T e946e56dc81e45a5ada9f1847889d2c2 P 9289839d25724333854785220983c1bb: No bootstrap required, opened a new log
I20260926 01:03:27.542397 22481 ts_tablet_manager.cc:1397] T e946e56dc81e45a5ada9f1847889d2c2 P 9289839d25724333854785220983c1bb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:03:27.544287 22481 raft_consensus.cc:348] T e946e56dc81e45a5ada9f1847889d2c2 P 9289839d25724333854785220983c1bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8e18660a84c4a89881e04dd110ba577" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 44585 } } peers { permanent_uuid: "9fb55caaa9474d899c980fdc9965eac6" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 44587 } } peers { permanent_uuid: "9289839d25724333854785220983c1bb" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 44757 } }
I20260926 01:03:27.544436 22481 raft_consensus.cc:374] T e946e56dc81e45a5ada9f1847889d2c2 P 9289839d25724333854785220983c1bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:03:27.544481 22481 raft_consensus.cc:729] T e946e56dc81e45a5ada9f1847889d2c2 P 9289839d25724333854785220983c1bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9289839d25724333854785220983c1bb, State: Initialized, Role: FOLLOWER
I20260926 01:03:27.544613 22481 consensus_queue.cc:260] T e946e56dc81e45a5ada9f1847889d2c2 P 9289839d25724333854785220983c1bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8e18660a84c4a89881e04dd110ba577" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 44585 } } peers { permanent_uuid: "9fb55caaa9474d899c980fdc9965eac6" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 44587 } } peers { permanent_uuid: "9289839d25724333854785220983c1bb" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 44757 } }
I20260926 01:03:27.544847 22481 ts_tablet_manager.cc:1428] T e946e56dc81e45a5ada9f1847889d2c2 P 9289839d25724333854785220983c1bb: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 01:03:27.544890 22462 heartbeater.cc:503] Master 127.3.75.62:34535 was elected leader, sending a full tablet report...
I20260926 01:03:27.573995 22484 raft_consensus.cc:482] T e946e56dc81e45a5ada9f1847889d2c2 P 9289839d25724333854785220983c1bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:03:27.574108 22484 raft_consensus.cc:504] T e946e56dc81e45a5ada9f1847889d2c2 P 9289839d25724333854785220983c1bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8e18660a84c4a89881e04dd110ba577" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 44585 } } peers { permanent_uuid: "9fb55caaa9474d899c980fdc9965eac6" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 44587 } } peers { permanent_uuid: "9289839d25724333854785220983c1bb" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 44757 } }
I20260926 01:03:27.574399 22484 leader_election.cc:288] T e946e56dc81e45a5ada9f1847889d2c2 P 9289839d25724333854785220983c1bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8e18660a84c4a89881e04dd110ba577 (127.3.75.1:44585), 9fb55caaa9474d899c980fdc9965eac6 (127.3.75.2:44587)
I20260926 01:03:27.577342 22473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:27.574680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.3:39173 (local address 127.3.75.1:44585)
0926 01:03:27.574763 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:27.574766 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:27.574775 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:03:27.574809 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:27.574810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:27.574819 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:27.574948 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:03:27.575133 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:27.575685 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:27.576630 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:27.576994 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:27.577048 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:27.577056 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:27.577218 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 01:03:27.577265 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 01:03:27.577345 22464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:27.574585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.1:44585 (local address 127.3.75.3:39173)
0926 01:03:27.574647 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:27.574659 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:27.574715 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:27.574963 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:27.574964 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:27.574965 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:03:27.575057 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:27.575059 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:27.575770 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:27.575771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:27.576541 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:27.576544 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:27.577076 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:27.577077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:27.577147 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:27.577150 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:03:27.577199 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 01:03:27.577250 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":15}
I20260926 01:03:27.577380 22475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:27.574773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.3:44643 (local address 127.3.75.2:44587)
0926 01:03:27.574834 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:27.574836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:27.575009 (+ 173us) server_negotiation.cc:407] Connection header received
0926 01:03:27.575049 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:27.575049 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:27.575058 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:27.575118 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:03:27.575273 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:27.575712 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:27.576666 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:27.577011 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:27.577096 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:27.577105 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:27.577268 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 01:03:27.577312 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 01:03:27.577370 22485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:27.574720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.2:44587 (local address 127.3.75.3:44643)
0926 01:03:27.574931 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:27.574941 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:27.575000 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:27.575126 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:27.575127 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:27.575128 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:03:27.575208 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:27.575210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:27.575813 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:27.575813 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:27.576551 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:27.576553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:27.577116 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:27.577117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:27.577174 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:27.577177 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:03:27.577248 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 01:03:27.577282 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":1,"thread_start_us":123,"threads_started":1}
I20260926 01:03:27.577735 22150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e946e56dc81e45a5ada9f1847889d2c2" candidate_uuid: "9289839d25724333854785220983c1bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8e18660a84c4a89881e04dd110ba577" is_pre_election: true
I20260926 01:03:27.577735 22283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e946e56dc81e45a5ada9f1847889d2c2" candidate_uuid: "9289839d25724333854785220983c1bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fb55caaa9474d899c980fdc9965eac6" is_pre_election: true
I20260926 01:03:27.577891 22283 raft_consensus.cc:2454] T e946e56dc81e45a5ada9f1847889d2c2 P 9fb55caaa9474d899c980fdc9965eac6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9289839d25724333854785220983c1bb in term 0.
I20260926 01:03:27.577891 22150 raft_consensus.cc:2454] T e946e56dc81e45a5ada9f1847889d2c2 P a8e18660a84c4a89881e04dd110ba577 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9289839d25724333854785220983c1bb in term 0.
I20260926 01:03:27.578099 22350 leader_election.cc:302] T e946e56dc81e45a5ada9f1847889d2c2 P 9289839d25724333854785220983c1bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9289839d25724333854785220983c1bb, 9fb55caaa9474d899c980fdc9965eac6; no voters:
I20260926 01:03:27.578250 22484 raft_consensus.cc:2781] T e946e56dc81e45a5ada9f1847889d2c2 P 9289839d25724333854785220983c1bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:03:27.578318 22484 raft_consensus.cc:482] T e946e56dc81e45a5ada9f1847889d2c2 P 9289839d25724333854785220983c1bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:03:27.578336 22484 raft_consensus.cc:3037] T e946e56dc81e45a5ada9f1847889d2c2 P 9289839d25724333854785220983c1bb [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:03:27.579180 22484 raft_consensus.cc:504] T e946e56dc81e45a5ada9f1847889d2c2 P 9289839d25724333854785220983c1bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8e18660a84c4a89881e04dd110ba577" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 44585 } } peers { permanent_uuid: "9fb55caaa9474d899c980fdc9965eac6" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 44587 } } peers { permanent_uuid: "9289839d25724333854785220983c1bb" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 44757 } }
I20260926 01:03:27.579337 22484 leader_election.cc:288] T e946e56dc81e45a5ada9f1847889d2c2 P 9289839d25724333854785220983c1bb [CANDIDATE]: Term 1 election: Requested vote from peers a8e18660a84c4a89881e04dd110ba577 (127.3.75.1:44585), 9fb55caaa9474d899c980fdc9965eac6 (127.3.75.2:44587)
I20260926 01:03:27.579469 22150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e946e56dc81e45a5ada9f1847889d2c2" candidate_uuid: "9289839d25724333854785220983c1bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8e18660a84c4a89881e04dd110ba577"
I20260926 01:03:27.579496 22283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e946e56dc81e45a5ada9f1847889d2c2" candidate_uuid: "9289839d25724333854785220983c1bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fb55caaa9474d899c980fdc9965eac6"
I20260926 01:03:27.579553 22283 raft_consensus.cc:3037] T e946e56dc81e45a5ada9f1847889d2c2 P 9fb55caaa9474d899c980fdc9965eac6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:03:27.579548 22150 raft_consensus.cc:3037] T e946e56dc81e45a5ada9f1847889d2c2 P a8e18660a84c4a89881e04dd110ba577 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:03:27.580418 22283 raft_consensus.cc:2454] T e946e56dc81e45a5ada9f1847889d2c2 P 9fb55caaa9474d899c980fdc9965eac6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9289839d25724333854785220983c1bb in term 1.
I20260926 01:03:27.580418 22150 raft_consensus.cc:2454] T e946e56dc81e45a5ada9f1847889d2c2 P a8e18660a84c4a89881e04dd110ba577 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9289839d25724333854785220983c1bb in term 1.
I20260926 01:03:27.580581 22350 leader_election.cc:302] T e946e56dc81e45a5ada9f1847889d2c2 P 9289839d25724333854785220983c1bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9289839d25724333854785220983c1bb, 9fb55caaa9474d899c980fdc9965eac6; no voters:
I20260926 01:03:27.580665 22484 raft_consensus.cc:2781] T e946e56dc81e45a5ada9f1847889d2c2 P 9289839d25724333854785220983c1bb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:03:27.580825 22484 raft_consensus.cc:686] T e946e56dc81e45a5ada9f1847889d2c2 P 9289839d25724333854785220983c1bb [term 1 LEADER]: Becoming Leader. State: Replica: 9289839d25724333854785220983c1bb, State: Running, Role: LEADER
I20260926 01:03:27.580919 22484 consensus_queue.cc:237] T e946e56dc81e45a5ada9f1847889d2c2 P 9289839d25724333854785220983c1bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8e18660a84c4a89881e04dd110ba577" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 44585 } } peers { permanent_uuid: "9fb55caaa9474d899c980fdc9965eac6" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 44587 } } peers { permanent_uuid: "9289839d25724333854785220983c1bb" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 44757 } }
I20260926 01:03:27.581670 22005 catalog_manager.cc:5495] T e946e56dc81e45a5ada9f1847889d2c2 P 9289839d25724333854785220983c1bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 9289839d25724333854785220983c1bb (127.3.75.3). New cstate: current_term: 1 leader_uuid: "9289839d25724333854785220983c1bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8e18660a84c4a89881e04dd110ba577" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 44585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fb55caaa9474d899c980fdc9965eac6" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 44587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9289839d25724333854785220983c1bb" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 44757 } health_report { overall_health: HEALTHY } } }
I20260926 01:03:27.595110 22472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:27.592793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58282 (local address 127.3.75.3:44757)
0926 01:03:27.592846 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:27.592848 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:27.592858 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:03:27.592899 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:27.592900 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:27.592917 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:27.592975 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:03:27.593080 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:27.593647 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:27.594386 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:27.594563 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:27.594611 (+ 48us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:27.594787 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:03:27.594909 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:03:27.594996 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:27.595032 (+ 36us) server_negotiation.cc:299] Negotiation successful
0926 01:03:27.595040 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 01:03:27.596966 22283 raft_consensus.cc:1261] T e946e56dc81e45a5ada9f1847889d2c2 P 9fb55caaa9474d899c980fdc9965eac6 [term 1 FOLLOWER]: Refusing update from remote peer 9289839d25724333854785220983c1bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:03:27.596966 22150 raft_consensus.cc:1261] T e946e56dc81e45a5ada9f1847889d2c2 P a8e18660a84c4a89881e04dd110ba577 [term 1 FOLLOWER]: Refusing update from remote peer 9289839d25724333854785220983c1bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:03:27.597316 22486 consensus_queue.cc:1035] T e946e56dc81e45a5ada9f1847889d2c2 P 9289839d25724333854785220983c1bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fb55caaa9474d899c980fdc9965eac6" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 44587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:03:27.597486 22484 consensus_queue.cc:1035] T e946e56dc81e45a5ada9f1847889d2c2 P 9289839d25724333854785220983c1bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8e18660a84c4a89881e04dd110ba577" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 44585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:03:27.599594 22489 mvcc.cc:205] Tried to move back new op lower bound from 7333415352714338304 to 7333415352652050432. Current Snapshot: MvccSnapshot[applied={T|T < 7333415352714338304}]
I20260926 01:03:27.599929 22491 mvcc.cc:205] Tried to move back new op lower bound from 7333415352714338304 to 7333415352652050432. Current Snapshot: MvccSnapshot[applied={T|T < 7333415352714338304}]
I20260926 01:03:27.600559 22490 mvcc.cc:205] Tried to move back new op lower bound from 7333415352714338304 to 7333415352652050432. Current Snapshot: MvccSnapshot[applied={T|T < 7333415352714338304 or (T in {7333415352714338304})}]
I20260926 01:03:27.602777 22006 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40018:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:03:27.602857 22006 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40018:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:03:27.603648 22006 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5eed5d13ee434329b69e8654b0777de9: Sending DeleteTablet for 3 replicas of tablet e946e56dc81e45a5ada9f1847889d2c2
I20260926 01:03:27.603974 22396 tablet_service.cc:1514] Processing DeleteTablet for tablet e946e56dc81e45a5ada9f1847889d2c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:03:27 UTC) from {username='slave'} at 127.0.0.1:58280
I20260926 01:03:27.604065 22263 tablet_service.cc:1514] Processing DeleteTablet for tablet e946e56dc81e45a5ada9f1847889d2c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:03:27 UTC) from {username='slave'} at 127.0.0.1:32892
I20260926 01:03:27.604071 22130 tablet_service.cc:1514] Processing DeleteTablet for tablet e946e56dc81e45a5ada9f1847889d2c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:03:27 UTC) from {username='slave'} at 127.0.0.1:39740
I20260926 01:03:27.604260 22499 tablet_replica.cc:330] T e946e56dc81e45a5ada9f1847889d2c2 P 9289839d25724333854785220983c1bb: stopping tablet replica
I20260926 01:03:27.604359 22499 raft_consensus.cc:2229] T e946e56dc81e45a5ada9f1847889d2c2 P 9289839d25724333854785220983c1bb [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:03:27.604447 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 22068
I20260926 01:03:27.604485 22499 raft_consensus.cc:2258] T e946e56dc81e45a5ada9f1847889d2c2 P 9289839d25724333854785220983c1bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:03:27.610860 22499 ts_tablet_manager.cc:1905] T e946e56dc81e45a5ada9f1847889d2c2 P 9289839d25724333854785220983c1bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:03:27.611032 22500 tablet_replica.cc:330] T e946e56dc81e45a5ada9f1847889d2c2 P 9fb55caaa9474d899c980fdc9965eac6: stopping tablet replica
I20260926 01:03:27.611172 22500 raft_consensus.cc:2229] T e946e56dc81e45a5ada9f1847889d2c2 P 9fb55caaa9474d899c980fdc9965eac6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:03:27.611349 22500 raft_consensus.cc:2258] T e946e56dc81e45a5ada9f1847889d2c2 P 9fb55caaa9474d899c980fdc9965eac6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:03:27.611843 22500 ts_tablet_manager.cc:1905] T e946e56dc81e45a5ada9f1847889d2c2 P 9fb55caaa9474d899c980fdc9965eac6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:03:27.612946 22499 ts_tablet_manager.cc:1918] T e946e56dc81e45a5ada9f1847889d2c2 P 9289839d25724333854785220983c1bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:03:27.613065 22499 log.cc:1196] T e946e56dc81e45a5ada9f1847889d2c2 P 9289839d25724333854785220983c1bb: Deleting WAL directory at /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/ts-2/wal/wals/e946e56dc81e45a5ada9f1847889d2c2
I20260926 01:03:27.613372 22499 ts_tablet_manager.cc:1939] T e946e56dc81e45a5ada9f1847889d2c2 P 9289839d25724333854785220983c1bb: Deleting consensus metadata
I20260926 01:03:27.613510 22500 ts_tablet_manager.cc:1918] T e946e56dc81e45a5ada9f1847889d2c2 P 9fb55caaa9474d899c980fdc9965eac6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:03:27.613677 22500 log.cc:1196] T e946e56dc81e45a5ada9f1847889d2c2 P 9fb55caaa9474d899c980fdc9965eac6: Deleting WAL directory at /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384372073861-3372-0/minicluster-data/ts-1/wal/wals/e946e56dc81e45a5ada9f1847889d2c2
I20260926 01:03:27.613852 21991 catalog_manager.cc:4841] TS 9289839d25724333854785220983c1bb (127.3.75.3:44757): tablet e946e56dc81e45a5ada9f1847889d2c2 (table test-table [id=39ac2420278348de8d3b4b6484eb5c6c]) successfully deleted
I20260926 01:03:27.613992 22500 ts_tablet_manager.cc:1939] T e946e56dc81e45a5ada9f1847889d2c2 P 9fb55caaa9474d899c980fdc9965eac6: Deleting consensus metadata
I20260926 01:03:27.614356 21993 catalog_manager.cc:4841] TS 9fb55caaa9474d899c980fdc9965eac6 (127.3.75.2:44587): tablet e946e56dc81e45a5ada9f1847889d2c2 (table test-table [id=39ac2420278348de8d3b4b6484eb5c6c]) successfully deleted
W20260926 01:03:27.615345 21993 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.75.1:44585 (error 108)
I20260926 01:03:27.615535 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 22200
I20260926 01:03:27.616247 22470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:27.615667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.1:44585 (local address 127.0.0.1:39748)
0926 01:03:27.616169 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:27.616210 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.1:44585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":478}
W20260926 01:03:27.617985 21993 catalog_manager.cc:4584] TS a8e18660a84c4a89881e04dd110ba577 (127.3.75.1:44585): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e946e56dc81e45a5ada9f1847889d2c2: Network error: Client connection negotiation failed: client connection to 127.3.75.1:44585: connect: Connection refused (error 111)
I20260926 01:03:27.622010 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 22333
I20260926 01:03:27.627101 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 21976
2026-09-26T01:03:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (659 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:03:27 dist-test-slave-7v10 krb5kdc[22505](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:03:27 dist-test-slave-7v10 krb5kdc[22505](info): set up 2 sockets
Sep 26 01:03:27 dist-test-slave-7v10 krb5kdc[22505](info): commencing operation
krb5kdc: starting...
W20260926 01:03:28.677644 3372 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Sep 26 01:03:28 dist-test-slave-7v10 krb5kdc[22505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:03:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:03:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.75.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab.
Entry for principal kudu/127.3.75.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab.
I20260926 01:03:28.792965 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:44719
--webserver_interface=127.3.75.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:40377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.75.62:44719
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:03:28.871932 22519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:03:28.872126 22519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:03:28.872148 22519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:03:28.873201 22519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:03:28.873237 22519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:03:28.873251 22519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:03:28.873265 22519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:03:28.873276 22519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:03:28.874624 22519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:40377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.75.62:44719
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:44719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:03:28.874812 22519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:03:28.874991 22519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:03:28.879544 22525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:03:28.879684 22519 server_base.cc:1034] running on GCE node
W20260926 01:03:28.879563 22524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:28.879570 22527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:03:28.880038 22519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:03:28.880307 22519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:03:28.881465 22519 hybrid_clock.cc:648] HybridClock initialized: now 1790384608881447 us; error 32 us; skew 500 ppm
Sep 26 01:03:28 dist-test-slave-7v10 krb5kdc[22505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384608, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:03:28.883881 22519 init.cc:377] Logged in from keytab as kudu/127.3.75.62@KRBTEST.COM (short username kudu)
I20260926 01:03:28.884330 22519 webserver.cc:460] Webserver started at http://127.3.75.62:44535/ using document root <none> and password file <none>
I20260926 01:03:28.884544 22519 fs_manager.cc:359] Metadata directory not provided
I20260926 01:03:28.884608 22519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:03:28.884754 22519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:03:28.885792 22519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/master-0/data/instance:
uuid: "5ab323b0faf7425aa021cb21a2883b95"
format_stamp: "Formatted at 2026-09-26 01:03:28 on dist-test-slave-7v10"
server_key: "eeecf403ac6de5670512f3e641ae5189"
server_key_iv: "b0b68d0ac8be12ef4387ffaf6def3fde"
server_key_version: "encryptionkey@0"
I20260926 01:03:28.886143 22519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/master-0/wal/instance:
uuid: "5ab323b0faf7425aa021cb21a2883b95"
format_stamp: "Formatted at 2026-09-26 01:03:28 on dist-test-slave-7v10"
server_key: "eeecf403ac6de5670512f3e641ae5189"
server_key_iv: "b0b68d0ac8be12ef4387ffaf6def3fde"
server_key_version: "encryptionkey@0"
I20260926 01:03:28.887580 22519 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:03:28.888343 22534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:28.888520 22519 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:03:28.888605 22519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/master-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/master-0/wal
uuid: "5ab323b0faf7425aa021cb21a2883b95"
format_stamp: "Formatted at 2026-09-26 01:03:28 on dist-test-slave-7v10"
server_key: "eeecf403ac6de5670512f3e641ae5189"
server_key_iv: "b0b68d0ac8be12ef4387ffaf6def3fde"
server_key_version: "encryptionkey@0"
I20260926 01:03:28.888659 22519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:03:28.899389 22519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:03:28.901081 22519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:03:28.901186 22519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:03:28.905025 22519 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.62:44719
I20260926 01:03:28.905071 22586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.62:44719 every 8 connection(s)
I20260926 01:03:28.905396 22519 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
I20260926 01:03:28.906184 22587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:03:28.907789 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 22519
I20260926 01:03:28.907920 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/master-0/wal/instance
I20260926 01:03:28.908123 3372 external_mini_cluster.cc:1428] Setting key c4c6de298647cf4d2f38d9cc6b847ba3
I20260926 01:03:28.908869 22587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ab323b0faf7425aa021cb21a2883b95: Bootstrap starting.
I20260926 01:03:28.909452 22587 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ab323b0faf7425aa021cb21a2883b95: Neither blocks nor log segments found. Creating new log.
I20260926 01:03:28.909786 22587 log.cc:824] T 00000000000000000000000000000000 P 5ab323b0faf7425aa021cb21a2883b95: Log is configured to *not* fsync() on all Append() calls
I20260926 01:03:28.910442 22587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ab323b0faf7425aa021cb21a2883b95: No bootstrap required, opened a new log
I20260926 01:03:28.912575 22587 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ab323b0faf7425aa021cb21a2883b95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ab323b0faf7425aa021cb21a2883b95" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 44719 } }
I20260926 01:03:28.912734 22587 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ab323b0faf7425aa021cb21a2883b95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:03:28.912768 22587 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ab323b0faf7425aa021cb21a2883b95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ab323b0faf7425aa021cb21a2883b95, State: Initialized, Role: FOLLOWER
I20260926 01:03:28.912977 22587 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ab323b0faf7425aa021cb21a2883b95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ab323b0faf7425aa021cb21a2883b95" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 44719 } }
I20260926 01:03:28.913074 22587 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ab323b0faf7425aa021cb21a2883b95 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:03:28.913100 22587 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ab323b0faf7425aa021cb21a2883b95 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:03:28.913164 22587 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ab323b0faf7425aa021cb21a2883b95 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:03:28.913892 22587 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ab323b0faf7425aa021cb21a2883b95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ab323b0faf7425aa021cb21a2883b95" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 44719 } }
I20260926 01:03:28.914012 22587 leader_election.cc:302] T 00000000000000000000000000000000 P 5ab323b0faf7425aa021cb21a2883b95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ab323b0faf7425aa021cb21a2883b95; no voters:
I20260926 01:03:28.914193 22587 leader_election.cc:288] T 00000000000000000000000000000000 P 5ab323b0faf7425aa021cb21a2883b95 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:03:28.914247 22592 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ab323b0faf7425aa021cb21a2883b95 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:03:28.914394 22592 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ab323b0faf7425aa021cb21a2883b95 [term 1 LEADER]: Becoming Leader. State: Replica: 5ab323b0faf7425aa021cb21a2883b95, State: Running, Role: LEADER
I20260926 01:03:28.914453 22587 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ab323b0faf7425aa021cb21a2883b95 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:03:28.914533 22592 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ab323b0faf7425aa021cb21a2883b95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ab323b0faf7425aa021cb21a2883b95" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 44719 } }
I20260926 01:03:28.914952 22592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ab323b0faf7425aa021cb21a2883b95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ab323b0faf7425aa021cb21a2883b95. Latest consensus state: current_term: 1 leader_uuid: "5ab323b0faf7425aa021cb21a2883b95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ab323b0faf7425aa021cb21a2883b95" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 44719 } } }
I20260926 01:03:28.915081 22592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ab323b0faf7425aa021cb21a2883b95 [sys.catalog]: This master's current role is: LEADER
Sep 26 01:03:28 dist-test-slave-7v10 krb5kdc[22505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 01:03:28.915170 22593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ab323b0faf7425aa021cb21a2883b95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ab323b0faf7425aa021cb21a2883b95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ab323b0faf7425aa021cb21a2883b95" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 44719 } } }
I20260926 01:03:28.915230 22593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ab323b0faf7425aa021cb21a2883b95 [sys.catalog]: This master's current role is: LEADER
W20260926 01:03:28.916764 22608 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5ab323b0faf7425aa021cb21a2883b95: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:03:28.916821 22608 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:03:28.916869 22599 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:03:28.917021 22599 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:03:28.918541 22599 catalog_manager.cc:1347] Generated new cluster ID: 72196d9807fc4cdfb543997bf484e907
I20260926 01:03:28.918593 22599 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:03:28.919628 22590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:28.908636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48604 (local address 127.3.75.62:44719)
0926 01:03:28.908811 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:28.908818 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:28.908843 (+ 25us) server_negotiation.cc:407] Connection header received
0926 01:03:28.909301 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:28.909308 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:28.909357 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:28.909467 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:28.910220 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:28.911176 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:28.912293 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:28.912656 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:28.912759 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:28.915763 (+ 3004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:28.915768 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:28.915768 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:03:28.915780 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:28.917718 (+ 1938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:28.918100 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:28.918103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:28.918104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:28.918206 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:28.918425 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:28.918426 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:28.918426 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:28.918738 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:28.918899 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:28.919326 (+ 427us) server_negotiation.cc:299] Negotiation successful
0926 01:03:28.919448 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":91,"threads_started":1}
I20260926 01:03:28.930460 22599 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:03:28.931252 22599 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:03:28.942359 22599 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ab323b0faf7425aa021cb21a2883b95: Generated new TSK 0
I20260926 01:03:28.942546 22599 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.75.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab.
Entry for principal kudu/127.3.75.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab.
I20260926 01:03:28.975494 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.1:0
--local_ip_for_outbound_sockets=127.3.75.1
--webserver_interface=127.3.75.1
--webserver_port=0
--tserver_master_addrs=127.3.75.62:44719
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab
--principal=kudu/127.3.75.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:40377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:03:29.080430 22613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:03:29.080633 22613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:03:29.080658 22613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:03:29.081801 22613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:03:29.081840 22613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:03:29.081866 22613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.1
I20260926 01:03:29.083343 22613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:40377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.1.keytab
--principal=kudu/127.3.75.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.75.1
--webserver_port=0
--tserver_master_addrs=127.3.75.62:44719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.1
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:03:29.083545 22613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:03:29.083720 22613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:03:29.088734 22621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:29.088734 22619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:29.088848 22618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:03:29.088869 22613 server_base.cc:1034] running on GCE node
I20260926 01:03:29.089293 22613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:03:29.089583 22613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:03:29.090755 22613 hybrid_clock.cc:648] HybridClock initialized: now 1790384609090723 us; error 49 us; skew 500 ppm
Sep 26 01:03:29 dist-test-slave-7v10 krb5kdc[22505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384609, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:03:29.093264 22613 init.cc:377] Logged in from keytab as kudu/127.3.75.1@KRBTEST.COM (short username kudu)
I20260926 01:03:29.093780 22613 webserver.cc:460] Webserver started at http://127.3.75.1:43555/ using document root <none> and password file <none>
I20260926 01:03:29.094017 22613 fs_manager.cc:359] Metadata directory not provided
I20260926 01:03:29.094080 22613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:03:29.094229 22613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:03:29.095340 22613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/ts-0/data/instance:
uuid: "0c35034b75cf40bd80f78d14645555f1"
format_stamp: "Formatted at 2026-09-26 01:03:29 on dist-test-slave-7v10"
server_key: "2c73179da7aa28ac8da22bb7bbae0a33"
server_key_iv: "fcf890d244bae6689e032634ef1a4c87"
server_key_version: "encryptionkey@0"
I20260926 01:03:29.095700 22613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/ts-0/wal/instance:
uuid: "0c35034b75cf40bd80f78d14645555f1"
format_stamp: "Formatted at 2026-09-26 01:03:29 on dist-test-slave-7v10"
server_key: "2c73179da7aa28ac8da22bb7bbae0a33"
server_key_iv: "fcf890d244bae6689e032634ef1a4c87"
server_key_version: "encryptionkey@0"
I20260926 01:03:29.096998 22613 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:03:29.097695 22628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:29.097869 22613 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:03:29.097962 22613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/ts-0/wal
uuid: "0c35034b75cf40bd80f78d14645555f1"
format_stamp: "Formatted at 2026-09-26 01:03:29 on dist-test-slave-7v10"
server_key: "2c73179da7aa28ac8da22bb7bbae0a33"
server_key_iv: "fcf890d244bae6689e032634ef1a4c87"
server_key_version: "encryptionkey@0"
I20260926 01:03:29.098037 22613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:03:29.110312 22613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:03:29.112185 22613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:03:29.112347 22613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:03:29.112565 22613 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:03:29.112875 22613 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:03:29.112926 22613 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:29.112959 22613 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:03:29.112986 22613 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:29.119091 22613 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.1:38395
I20260926 01:03:29.119143 22742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.1:38395 every 8 connection(s)
I20260926 01:03:29.119477 22613 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
I20260926 01:03:29.120954 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 22613
I20260926 01:03:29.121095 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/ts-0/wal/instance
I20260926 01:03:29.121302 3372 external_mini_cluster.cc:1428] Setting key 06593db78d800286a788019d91842019
Sep 26 01:03:29 dist-test-slave-7v10 krb5kdc[22505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384609, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.1@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 01:03:29.129357 22590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:29.120189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:50409 (local address 127.3.75.62:44719)
0926 01:03:29.120256 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:29.120258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:29.120478 (+ 220us) server_negotiation.cc:407] Connection header received
0926 01:03:29.120997 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:29.120999 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:29.121011 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:29.121059 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:29.122148 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:29.122662 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:29.123730 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:29.123949 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:29.124031 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:29.126395 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:29.126396 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:29.126396 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:03:29.126403 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:29.127896 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:29.128380 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:29.128380 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:29.128381 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:29.128428 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:29.128723 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:29.128723 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:29.128723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:29.128824 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:29.128873 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:29.129216 (+ 343us) server_negotiation.cc:299] Negotiation successful
0926 01:03:29.129289 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
WARNING: no policy specified for kudu/127.3.75.2@KRBTEST.COM; defaulting to no policy
I20260926 01:03:29.129638 22745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:29.120200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44719 (local address 127.3.75.1:50409)
0926 01:03:29.120358 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:29.120393 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:29.120892 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:29.121141 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:29.121145 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:29.121796 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:03:29.122049 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:29.122056 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:29.122756 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:29.122757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:29.123615 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:29.123616 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:29.124042 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:29.124043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:29.124176 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:29.124705 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:03:29.124713 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:03:29.126294 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:03:29.127981 (+ 1687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:03:29.127988 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:03:29.127991 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:03:29.128301 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:03:29.128491 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:03:29.128492 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:03:29.128492 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:03:29.128644 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:03:29.128889 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:03:29.128898 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:03:29.129081 (+ 183us) client_negotiation.cc:769] Sending connection context
0926 01:03:29.129201 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 01:03:29.129373 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":71,"threads_started":1}
I20260926 01:03:29.130160 22743 heartbeater.cc:347] Connected to a master server at 127.3.75.62:44719
I20260926 01:03:29.130242 22743 heartbeater.cc:464] Registering TS with master...
I20260926 01:03:29.130390 22743 heartbeater.cc:511] Master 127.3.75.62:44719 requested a full tablet report, sending...
I20260926 01:03:29.130930 22551 ts_manager.cc:194] Registered new tserver with Master: 0c35034b75cf40bd80f78d14645555f1 (127.3.75.1:38395)
I20260926 01:03:29.131654 22551 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.75.1@KRBTEST.COM'} at 127.3.75.1:50409
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab.
Entry for principal kudu/127.3.75.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab.
I20260926 01:03:29.145824 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.2:0
--local_ip_for_outbound_sockets=127.3.75.2
--webserver_interface=127.3.75.2
--webserver_port=0
--tserver_master_addrs=127.3.75.62:44719
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab
--principal=kudu/127.3.75.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:40377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:03:29.229010 22748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:03:29.229212 22748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:03:29.229236 22748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:03:29.230247 22748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:03:29.230279 22748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:03:29.230304 22748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.2
I20260926 01:03:29.231762 22748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:40377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.2.keytab
--principal=kudu/127.3.75.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.75.2
--webserver_port=0
--tserver_master_addrs=127.3.75.62:44719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.2
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:03:29.231945 22748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:03:29.232111 22748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:03:29.237082 22754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:29.237085 22756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:29.237072 22753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:03:29.237239 22748 server_base.cc:1034] running on GCE node
I20260926 01:03:29.237488 22748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:03:29.237753 22748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:03:29.238961 22748 hybrid_clock.cc:648] HybridClock initialized: now 1790384609238936 us; error 38 us; skew 500 ppm
Sep 26 01:03:29 dist-test-slave-7v10 krb5kdc[22505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384609, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:03:29.241514 22748 init.cc:377] Logged in from keytab as kudu/127.3.75.2@KRBTEST.COM (short username kudu)
I20260926 01:03:29.241882 22748 webserver.cc:460] Webserver started at http://127.3.75.2:41397/ using document root <none> and password file <none>
I20260926 01:03:29.242079 22748 fs_manager.cc:359] Metadata directory not provided
I20260926 01:03:29.242118 22748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:03:29.242203 22748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:03:29.243247 22748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/ts-1/data/instance:
uuid: "6feefb0d2fee41c69f2dd1540810307e"
format_stamp: "Formatted at 2026-09-26 01:03:29 on dist-test-slave-7v10"
server_key: "5b09315774dd72fac33089705bba9017"
server_key_iv: "9a4ac242b2d73f8ce0dd286db7504810"
server_key_version: "encryptionkey@0"
I20260926 01:03:29.243536 22748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/ts-1/wal/instance:
uuid: "6feefb0d2fee41c69f2dd1540810307e"
format_stamp: "Formatted at 2026-09-26 01:03:29 on dist-test-slave-7v10"
server_key: "5b09315774dd72fac33089705bba9017"
server_key_iv: "9a4ac242b2d73f8ce0dd286db7504810"
server_key_version: "encryptionkey@0"
I20260926 01:03:29.244779 22748 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:03:29.245448 22763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:29.245644 22748 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:03:29.245713 22748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/ts-1/wal
uuid: "6feefb0d2fee41c69f2dd1540810307e"
format_stamp: "Formatted at 2026-09-26 01:03:29 on dist-test-slave-7v10"
server_key: "5b09315774dd72fac33089705bba9017"
server_key_iv: "9a4ac242b2d73f8ce0dd286db7504810"
server_key_version: "encryptionkey@0"
I20260926 01:03:29.245822 22748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:03:29.258471 22748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:03:29.260779 22748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:03:29.260943 22748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:03:29.261176 22748 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:03:29.261482 22748 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:03:29.261538 22748 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:29.261580 22748 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:03:29.261607 22748 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:29.267118 22748 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.2:41559
I20260926 01:03:29.267197 22877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.2:41559 every 8 connection(s)
I20260926 01:03:29.267511 22748 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
I20260926 01:03:29.270505 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 22748
I20260926 01:03:29.270641 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/ts-1/wal/instance
I20260926 01:03:29.270910 3372 external_mini_cluster.cc:1428] Setting key 71231b7d5ef758d0e91aa35a7190ba3d
Sep 26 01:03:29 dist-test-slave-7v10 krb5kdc[22505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384609, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.2@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 01:03:29.277722 22590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:29.268278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:45577 (local address 127.3.75.62:44719)
0926 01:03:29.268322 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:29.268324 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:29.268518 (+ 194us) server_negotiation.cc:407] Connection header received
0926 01:03:29.268967 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:29.268967 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:29.268976 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:29.269039 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:29.269908 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:29.270491 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:29.271863 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:29.272165 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:29.272251 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:29.274557 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:29.274558 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:29.274558 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:03:29.274566 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:29.276313 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:29.276732 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:29.276734 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:29.276734 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:29.276799 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:29.277035 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:29.277036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:29.277036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:29.277188 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:29.277268 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:29.277519 (+ 251us) server_negotiation.cc:299] Negotiation successful
0926 01:03:29.277609 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 01:03:29.277740 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:29.268230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44719 (local address 127.3.75.2:45577)
0926 01:03:29.268388 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:29.268428 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:29.268890 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:29.269090 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:29.269093 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:29.269630 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:03:29.269824 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:29.269840 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:29.270611 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:29.270612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:29.271783 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:29.271786 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:29.272258 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:29.272258 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:29.272356 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:29.272889 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:03:29.272897 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:03:29.274461 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:03:29.276409 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:03:29.276416 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:03:29.276419 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:03:29.276641 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:03:29.276871 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:03:29.276871 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:03:29.276872 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:03:29.276965 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:03:29.277258 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:03:29.277262 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:03:29.277384 (+ 122us) client_negotiation.cc:769] Sending connection context
0926 01:03:29.277465 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 01:03:29.277582 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":91,"threads_started":1}
I20260926 01:03:29.278291 22878 heartbeater.cc:347] Connected to a master server at 127.3.75.62:44719
I20260926 01:03:29.278403 22878 heartbeater.cc:464] Registering TS with master...
I20260926 01:03:29.278600 22878 heartbeater.cc:511] Master 127.3.75.62:44719 requested a full tablet report, sending...
I20260926 01:03:29.279009 22551 ts_manager.cc:194] Registered new tserver with Master: 6feefb0d2fee41c69f2dd1540810307e (127.3.75.2:41559)
I20260926 01:03:29.279412 22551 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.75.2@KRBTEST.COM'} at 127.3.75.2:45577
WARNING: no policy specified for kudu/127.3.75.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab.
Entry for principal kudu/127.3.75.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab.
I20260926 01:03:29.297859 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.3:0
--local_ip_for_outbound_sockets=127.3.75.3
--webserver_interface=127.3.75.3
--webserver_port=0
--tserver_master_addrs=127.3.75.62:44719
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab
--principal=kudu/127.3.75.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:40377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:03:29.379134 22883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:03:29.379418 22883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:03:29.379459 22883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:03:29.380633 22883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:03:29.380717 22883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:03:29.380765 22883 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.3
I20260926 01:03:29.382262 22883 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:40377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.3.keytab
--principal=kudu/127.3.75.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.75.3
--webserver_port=0
--tserver_master_addrs=127.3.75.62:44719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.3
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:03:29.382560 22883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:03:29.382805 22883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:03:29.388360 22889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:29.388361 22891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:29.388438 22888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:03:29.388378 22883 server_base.cc:1034] running on GCE node
I20260926 01:03:29.388849 22883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:03:29.389087 22883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:03:29.390260 22883 hybrid_clock.cc:648] HybridClock initialized: now 1790384609390227 us; error 48 us; skew 500 ppm
Sep 26 01:03:29 dist-test-slave-7v10 krb5kdc[22505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384609, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:03:29.393005 22883 init.cc:377] Logged in from keytab as kudu/127.3.75.3@KRBTEST.COM (short username kudu)
I20260926 01:03:29.393447 22883 webserver.cc:460] Webserver started at http://127.3.75.3:41583/ using document root <none> and password file <none>
I20260926 01:03:29.393689 22883 fs_manager.cc:359] Metadata directory not provided
I20260926 01:03:29.393759 22883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:03:29.393906 22883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:03:29.394999 22883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/ts-2/data/instance:
uuid: "1112613e1b9142908b4b0c3091e420c2"
format_stamp: "Formatted at 2026-09-26 01:03:29 on dist-test-slave-7v10"
server_key: "7343ad933e28819ec007e070d269068c"
server_key_iv: "9f82e35d97e7c30922015f288c41f9f2"
server_key_version: "encryptionkey@0"
I20260926 01:03:29.395359 22883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/ts-2/wal/instance:
uuid: "1112613e1b9142908b4b0c3091e420c2"
format_stamp: "Formatted at 2026-09-26 01:03:29 on dist-test-slave-7v10"
server_key: "7343ad933e28819ec007e070d269068c"
server_key_iv: "9f82e35d97e7c30922015f288c41f9f2"
server_key_version: "encryptionkey@0"
I20260926 01:03:29.396622 22883 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:03:29.397298 22898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:29.397461 22883 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:03:29.397564 22883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/ts-2/wal
uuid: "1112613e1b9142908b4b0c3091e420c2"
format_stamp: "Formatted at 2026-09-26 01:03:29 on dist-test-slave-7v10"
server_key: "7343ad933e28819ec007e070d269068c"
server_key_iv: "9f82e35d97e7c30922015f288c41f9f2"
server_key_version: "encryptionkey@0"
I20260926 01:03:29.397641 22883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:03:29.417032 22883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:03:29.419385 22883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:03:29.419544 22883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:03:29.419760 22883 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:03:29.420061 22883 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:03:29.420120 22883 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:29.420169 22883 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:03:29.420198 22883 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:29.425791 22883 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.3:37463
I20260926 01:03:29.425859 23012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.3:37463 every 8 connection(s)
I20260926 01:03:29.426187 22883 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
Sep 26 01:03:29 dist-test-slave-7v10 krb5kdc[22505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384609, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.3@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 01:03:29.432688 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 22883
I20260926 01:03:29.432787 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/ts-2/wal/instance
I20260926 01:03:29.432967 3372 external_mini_cluster.cc:1428] Setting key 596987b91402abb4ea2dca5af8432ca6
I20260926 01:03:29.435304 22590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:29.426935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.3:34779 (local address 127.3.75.62:44719)
0926 01:03:29.427006 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:29.427007 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:29.427222 (+ 215us) server_negotiation.cc:407] Connection header received
0926 01:03:29.427691 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:29.427692 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:29.427702 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:29.427763 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:29.428643 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:29.429149 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:29.430109 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:29.430316 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:29.430364 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:29.432402 (+ 2038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:29.432403 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:29.432404 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:03:29.432410 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:29.434118 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:29.434467 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:29.434468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:29.434468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:29.434516 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:29.434686 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:29.434687 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:29.434687 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:29.434783 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:29.434833 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:29.435144 (+ 311us) server_negotiation.cc:299] Negotiation successful
0926 01:03:29.435229 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 01:03:29.435472 23015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:29.426912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44719 (local address 127.3.75.3:34779)
0926 01:03:29.427091 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:29.427132 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:29.427602 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:29.427817 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:29.427820 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:29.428379 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:03:29.428569 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:29.428574 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:29.429235 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:29.429236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:29.430036 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:29.430038 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:29.430388 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:29.430389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:29.430491 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:29.430985 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:03:29.430994 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:03:29.432316 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:03:29.434186 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:03:29.434192 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:03:29.434194 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:03:29.434402 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:03:29.434555 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:03:29.434556 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:03:29.434556 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:03:29.434635 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:03:29.434840 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:03:29.434844 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:03:29.435019 (+ 175us) client_negotiation.cc:769] Sending connection context
0926 01:03:29.435104 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 01:03:29.435261 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":108,"threads_started":1}
I20260926 01:03:29.435995 23013 heartbeater.cc:347] Connected to a master server at 127.3.75.62:44719
I20260926 01:03:29.436100 23013 heartbeater.cc:464] Registering TS with master...
I20260926 01:03:29.436273 23013 heartbeater.cc:511] Master 127.3.75.62:44719 requested a full tablet report, sending...
I20260926 01:03:29.436636 22551 ts_manager.cc:194] Registered new tserver with Master: 1112613e1b9142908b4b0c3091e420c2 (127.3.75.3:37463)
I20260926 01:03:29.437014 22551 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.75.3@KRBTEST.COM'} at 127.3.75.3:34779
I20260926 01:03:29.444621 3372 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 01:03:29.451556 22590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:29.445931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48612 (local address 127.3.75.62:44719)
0926 01:03:29.445999 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:29.446001 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:29.446070 (+ 69us) server_negotiation.cc:407] Connection header received
0926 01:03:29.446106 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:29.446107 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:29.446115 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:29.446164 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:29.446686 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:29.447209 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:29.447952 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:29.448140 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:29.448189 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:29.449415 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:29.449416 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:29.449416 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:03:29.449426 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:29.450591 (+ 1165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:29.450939 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:29.450940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:29.450940 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:29.450990 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:29.451135 (+ 145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:29.451135 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:29.451136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:29.451262 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:29.451335 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:29.451436 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 01:03:29.451496 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 01:03:29.453172 22551 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48612:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:03:29.454717 22551 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:03:29.461158 23030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:29.457398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60280 (local address 127.3.75.3:37463)
0926 01:03:29.457555 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:29.457559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:29.457840 (+ 281us) server_negotiation.cc:407] Connection header received
0926 01:03:29.457881 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:29.457883 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:29.457898 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:29.457964 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:03:29.458162 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:29.459162 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:29.460246 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:29.460728 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:29.460828 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:29.460888 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:29.461038 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 01:03:29.461080 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":101,"threads_started":1}
I20260926 01:03:29.461145 23028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:29.457214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.3:37463 (local address 127.0.0.1:60280)
0926 01:03:29.457727 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:29.457737 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:29.457783 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:29.458036 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:29.458037 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:29.458038 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:03:29.458076 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:29.458077 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:29.459290 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:29.459291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:29.460152 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:29.460155 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:29.460832 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:29.460833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:29.460910 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:29.460930 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 01:03:29.461021 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 01:03:29.461073 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"mutex_wait_us":1,"thread_start_us":59,"threads_started":1}
I20260926 01:03:29.461351 23025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:29.457028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.1:38395 (local address 127.0.0.1:56484)
0926 01:03:29.457162 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:29.457204 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:29.457274 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:29.457743 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:29.457744 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:29.457747 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:03:29.457806 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:29.457808 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:29.458813 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:29.458814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:29.460190 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:29.460192 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:29.461091 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:29.461093 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:29.461183 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:29.461185 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 01:03:29.461226 (+ 41us) client_negotiation.cc:240] Negotiation successful
0926 01:03:29.461257 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"mutex_wait_us":61,"thread_start_us":85,"threads_started":1}
I20260926 01:03:29.461351 23026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:29.457028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.2:41559 (local address 127.0.0.1:60128)
0926 01:03:29.457390 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:29.457406 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:29.457447 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:29.457626 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:29.457629 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:29.457630 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:03:29.457736 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:29.457738 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:29.459002 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:29.459003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:29.460189 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:29.460192 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:29.461094 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:29.461094 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:29.461176 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:29.461181 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 01:03:29.461214 (+ 33us) client_negotiation.cc:240] Negotiation successful
0926 01:03:29.461256 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":6,"thread_start_us":45,"threads_started":1}
I20260926 01:03:29.461630 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:29.457216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60128 (local address 127.3.75.2:41559)
0926 01:03:29.457437 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:29.457442 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:29.457453 (+ 11us) server_negotiation.cc:407] Connection header received
0926 01:03:29.457514 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:29.457516 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:29.457532 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:29.457620 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:03:29.457821 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:29.458902 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:29.460294 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:29.460989 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:29.461383 (+ 394us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:29.461457 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:29.461484 (+ 27us) server_negotiation.cc:299] Negotiation successful
0926 01:03:29.461533 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":82,"threads_started":1}
I20260926 01:03:29.461817 23027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:29.457155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56484 (local address 127.3.75.1:38395)
0926 01:03:29.457557 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:29.457562 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:29.457571 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:03:29.457614 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:29.457615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:29.457630 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:29.457881 (+ 251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:03:29.457954 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:29.458703 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:29.460294 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:29.460989 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:29.461610 (+ 621us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:29.461674 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:29.461700 (+ 26us) server_negotiation.cc:299] Negotiation successful
0926 01:03:29.461738 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":90,"threads_started":1}
I20260926 01:03:29.462313 22812 tablet_service.cc:1467] Processing CreateTablet for tablet f2cdbe5f94184cb085271c1114671ec4 (DEFAULT_TABLE table=test-table [id=c07b65ab604244deba7e5b24a287dd3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:03:29.462314 22946 tablet_service.cc:1467] Processing CreateTablet for tablet f2cdbe5f94184cb085271c1114671ec4 (DEFAULT_TABLE table=test-table [id=c07b65ab604244deba7e5b24a287dd3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:03:29.462440 22677 tablet_service.cc:1467] Processing CreateTablet for tablet f2cdbe5f94184cb085271c1114671ec4 (DEFAULT_TABLE table=test-table [id=c07b65ab604244deba7e5b24a287dd3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:03:29.462664 22812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2cdbe5f94184cb085271c1114671ec4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:03:29.462664 22946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2cdbe5f94184cb085271c1114671ec4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:03:29.462823 22677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2cdbe5f94184cb085271c1114671ec4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:03:29.465376 23031 tablet_bootstrap.cc:492] T f2cdbe5f94184cb085271c1114671ec4 P 0c35034b75cf40bd80f78d14645555f1: Bootstrap starting.
I20260926 01:03:29.465900 23033 tablet_bootstrap.cc:492] T f2cdbe5f94184cb085271c1114671ec4 P 6feefb0d2fee41c69f2dd1540810307e: Bootstrap starting.
I20260926 01:03:29.466065 23031 tablet_bootstrap.cc:654] T f2cdbe5f94184cb085271c1114671ec4 P 0c35034b75cf40bd80f78d14645555f1: Neither blocks nor log segments found. Creating new log.
I20260926 01:03:29.466084 23032 tablet_bootstrap.cc:492] T f2cdbe5f94184cb085271c1114671ec4 P 1112613e1b9142908b4b0c3091e420c2: Bootstrap starting.
I20260926 01:03:29.466693 23031 log.cc:824] T f2cdbe5f94184cb085271c1114671ec4 P 0c35034b75cf40bd80f78d14645555f1: Log is configured to *not* fsync() on all Append() calls
I20260926 01:03:29.466768 23032 tablet_bootstrap.cc:654] T f2cdbe5f94184cb085271c1114671ec4 P 1112613e1b9142908b4b0c3091e420c2: Neither blocks nor log segments found. Creating new log.
I20260926 01:03:29.467348 23032 log.cc:824] T f2cdbe5f94184cb085271c1114671ec4 P 1112613e1b9142908b4b0c3091e420c2: Log is configured to *not* fsync() on all Append() calls
I20260926 01:03:29.467518 23033 tablet_bootstrap.cc:654] T f2cdbe5f94184cb085271c1114671ec4 P 6feefb0d2fee41c69f2dd1540810307e: Neither blocks nor log segments found. Creating new log.
I20260926 01:03:29.467535 23031 tablet_bootstrap.cc:492] T f2cdbe5f94184cb085271c1114671ec4 P 0c35034b75cf40bd80f78d14645555f1: No bootstrap required, opened a new log
I20260926 01:03:29.467598 23031 ts_tablet_manager.cc:1397] T f2cdbe5f94184cb085271c1114671ec4 P 0c35034b75cf40bd80f78d14645555f1: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 01:03:29.467999 23033 log.cc:824] T f2cdbe5f94184cb085271c1114671ec4 P 6feefb0d2fee41c69f2dd1540810307e: Log is configured to *not* fsync() on all Append() calls
I20260926 01:03:29.468756 23033 tablet_bootstrap.cc:492] T f2cdbe5f94184cb085271c1114671ec4 P 6feefb0d2fee41c69f2dd1540810307e: No bootstrap required, opened a new log
I20260926 01:03:29.468832 23033 ts_tablet_manager.cc:1397] T f2cdbe5f94184cb085271c1114671ec4 P 6feefb0d2fee41c69f2dd1540810307e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 01:03:29.469547 23031 raft_consensus.cc:348] T f2cdbe5f94184cb085271c1114671ec4 P 0c35034b75cf40bd80f78d14645555f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c35034b75cf40bd80f78d14645555f1" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 38395 } } peers { permanent_uuid: "6feefb0d2fee41c69f2dd1540810307e" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41559 } } peers { permanent_uuid: "1112613e1b9142908b4b0c3091e420c2" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 37463 } }
I20260926 01:03:29.469727 23031 raft_consensus.cc:374] T f2cdbe5f94184cb085271c1114671ec4 P 0c35034b75cf40bd80f78d14645555f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:03:29.469765 23031 raft_consensus.cc:729] T f2cdbe5f94184cb085271c1114671ec4 P 0c35034b75cf40bd80f78d14645555f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c35034b75cf40bd80f78d14645555f1, State: Initialized, Role: FOLLOWER
I20260926 01:03:29.469884 23031 consensus_queue.cc:260] T f2cdbe5f94184cb085271c1114671ec4 P 0c35034b75cf40bd80f78d14645555f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c35034b75cf40bd80f78d14645555f1" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 38395 } } peers { permanent_uuid: "6feefb0d2fee41c69f2dd1540810307e" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41559 } } peers { permanent_uuid: "1112613e1b9142908b4b0c3091e420c2" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 37463 } }
I20260926 01:03:29.470184 23031 ts_tablet_manager.cc:1428] T f2cdbe5f94184cb085271c1114671ec4 P 0c35034b75cf40bd80f78d14645555f1: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260926 01:03:29.470191 22743 heartbeater.cc:503] Master 127.3.75.62:44719 was elected leader, sending a full tablet report...
I20260926 01:03:29.470786 23032 tablet_bootstrap.cc:492] T f2cdbe5f94184cb085271c1114671ec4 P 1112613e1b9142908b4b0c3091e420c2: No bootstrap required, opened a new log
I20260926 01:03:29.470861 23032 ts_tablet_manager.cc:1397] T f2cdbe5f94184cb085271c1114671ec4 P 1112613e1b9142908b4b0c3091e420c2: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260926 01:03:29.470909 23033 raft_consensus.cc:348] T f2cdbe5f94184cb085271c1114671ec4 P 6feefb0d2fee41c69f2dd1540810307e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c35034b75cf40bd80f78d14645555f1" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 38395 } } peers { permanent_uuid: "6feefb0d2fee41c69f2dd1540810307e" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41559 } } peers { permanent_uuid: "1112613e1b9142908b4b0c3091e420c2" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 37463 } }
I20260926 01:03:29.471071 23033 raft_consensus.cc:374] T f2cdbe5f94184cb085271c1114671ec4 P 6feefb0d2fee41c69f2dd1540810307e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:03:29.471150 23033 raft_consensus.cc:729] T f2cdbe5f94184cb085271c1114671ec4 P 6feefb0d2fee41c69f2dd1540810307e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6feefb0d2fee41c69f2dd1540810307e, State: Initialized, Role: FOLLOWER
I20260926 01:03:29.471295 23033 consensus_queue.cc:260] T f2cdbe5f94184cb085271c1114671ec4 P 6feefb0d2fee41c69f2dd1540810307e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c35034b75cf40bd80f78d14645555f1" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 38395 } } peers { permanent_uuid: "6feefb0d2fee41c69f2dd1540810307e" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41559 } } peers { permanent_uuid: "1112613e1b9142908b4b0c3091e420c2" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 37463 } }
I20260926 01:03:29.471570 23033 ts_tablet_manager.cc:1428] T f2cdbe5f94184cb085271c1114671ec4 P 6feefb0d2fee41c69f2dd1540810307e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 01:03:29.471597 22878 heartbeater.cc:503] Master 127.3.75.62:44719 was elected leader, sending a full tablet report...
I20260926 01:03:29.472522 23032 raft_consensus.cc:348] T f2cdbe5f94184cb085271c1114671ec4 P 1112613e1b9142908b4b0c3091e420c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c35034b75cf40bd80f78d14645555f1" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 38395 } } peers { permanent_uuid: "6feefb0d2fee41c69f2dd1540810307e" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41559 } } peers { permanent_uuid: "1112613e1b9142908b4b0c3091e420c2" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 37463 } }
I20260926 01:03:29.472687 23032 raft_consensus.cc:374] T f2cdbe5f94184cb085271c1114671ec4 P 1112613e1b9142908b4b0c3091e420c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:03:29.472720 23032 raft_consensus.cc:729] T f2cdbe5f94184cb085271c1114671ec4 P 1112613e1b9142908b4b0c3091e420c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1112613e1b9142908b4b0c3091e420c2, State: Initialized, Role: FOLLOWER
I20260926 01:03:29.472833 23032 consensus_queue.cc:260] T f2cdbe5f94184cb085271c1114671ec4 P 1112613e1b9142908b4b0c3091e420c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c35034b75cf40bd80f78d14645555f1" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 38395 } } peers { permanent_uuid: "6feefb0d2fee41c69f2dd1540810307e" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41559 } } peers { permanent_uuid: "1112613e1b9142908b4b0c3091e420c2" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 37463 } }
I20260926 01:03:29.473042 23032 ts_tablet_manager.cc:1428] T f2cdbe5f94184cb085271c1114671ec4 P 1112613e1b9142908b4b0c3091e420c2: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 01:03:29.473096 23013 heartbeater.cc:503] Master 127.3.75.62:44719 was elected leader, sending a full tablet report...
I20260926 01:03:29.482966 23039 raft_consensus.cc:482] T f2cdbe5f94184cb085271c1114671ec4 P 1112613e1b9142908b4b0c3091e420c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:03:29.483037 23039 raft_consensus.cc:504] T f2cdbe5f94184cb085271c1114671ec4 P 1112613e1b9142908b4b0c3091e420c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c35034b75cf40bd80f78d14645555f1" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 38395 } } peers { permanent_uuid: "6feefb0d2fee41c69f2dd1540810307e" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41559 } } peers { permanent_uuid: "1112613e1b9142908b4b0c3091e420c2" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 37463 } }
I20260926 01:03:29.483361 23039 leader_election.cc:288] T f2cdbe5f94184cb085271c1114671ec4 P 1112613e1b9142908b4b0c3091e420c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c35034b75cf40bd80f78d14645555f1 (127.3.75.1:38395), 6feefb0d2fee41c69f2dd1540810307e (127.3.75.2:41559)
I20260926 01:03:29.487035 23027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:29.483542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.3:45067 (local address 127.3.75.1:38395)
0926 01:03:29.483582 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:29.483584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:29.483634 (+ 50us) server_negotiation.cc:407] Connection header received
0926 01:03:29.483731 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:29.483732 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:29.483744 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:29.483800 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:03:29.483970 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:29.484672 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:29.486072 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:29.486659 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:29.486734 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:29.486743 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:29.486946 (+ 203us) server_negotiation.cc:299] Negotiation successful
0926 01:03:29.486977 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 01:03:29.487035 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:29.483552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.3:38125 (local address 127.3.75.2:41559)
0926 01:03:29.483602 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:29.483604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:29.483730 (+ 126us) server_negotiation.cc:407] Connection header received
0926 01:03:29.483802 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:29.483803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:29.483812 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:29.483875 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:03:29.484015 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:29.484677 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:29.486072 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:29.486668 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:29.486734 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:29.486743 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:29.486935 (+ 192us) server_negotiation.cc:299] Negotiation successful
0926 01:03:29.486972 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 01:03:29.487078 23015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:29.483478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.1:38395 (local address 127.3.75.3:45067)
0926 01:03:29.483577 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:29.483588 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:29.483636 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:29.483804 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:29.483805 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:29.483806 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:03:29.483902 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:29.483903 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:29.484753 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:29.484754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:29.485981 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:29.485984 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:29.486748 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:29.486749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:29.486840 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:29.486844 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:03:29.486939 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 01:03:29.486973 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"mutex_wait_us":7}
I20260926 01:03:29.487093 23040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:29.483513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.2:41559 (local address 127.3.75.3:38125)
0926 01:03:29.483645 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:29.483654 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:29.483744 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:29.483877 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:29.483878 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:29.483879 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:03:29.483958 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:29.483960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:29.484753 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:29.484754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:29.485981 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:29.485984 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:29.486748 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:29.486749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:29.486840 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:29.486844 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:03:29.486930 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 01:03:29.486973 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":66,"thread_start_us":79,"threads_started":1}
I20260926 01:03:29.487493 22832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2cdbe5f94184cb085271c1114671ec4" candidate_uuid: "1112613e1b9142908b4b0c3091e420c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6feefb0d2fee41c69f2dd1540810307e" is_pre_election: true
I20260926 01:03:29.487607 22832 raft_consensus.cc:2454] T f2cdbe5f94184cb085271c1114671ec4 P 6feefb0d2fee41c69f2dd1540810307e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1112613e1b9142908b4b0c3091e420c2 in term 0.
I20260926 01:03:29.487620 22697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2cdbe5f94184cb085271c1114671ec4" candidate_uuid: "1112613e1b9142908b4b0c3091e420c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c35034b75cf40bd80f78d14645555f1" is_pre_election: true
I20260926 01:03:29.487771 22697 raft_consensus.cc:2454] T f2cdbe5f94184cb085271c1114671ec4 P 0c35034b75cf40bd80f78d14645555f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1112613e1b9142908b4b0c3091e420c2 in term 0.
I20260926 01:03:29.487876 22900 leader_election.cc:302] T f2cdbe5f94184cb085271c1114671ec4 P 1112613e1b9142908b4b0c3091e420c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1112613e1b9142908b4b0c3091e420c2, 6feefb0d2fee41c69f2dd1540810307e; no voters:
I20260926 01:03:29.487987 23039 raft_consensus.cc:2781] T f2cdbe5f94184cb085271c1114671ec4 P 1112613e1b9142908b4b0c3091e420c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:03:29.488046 23039 raft_consensus.cc:482] T f2cdbe5f94184cb085271c1114671ec4 P 1112613e1b9142908b4b0c3091e420c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:03:29.488071 23039 raft_consensus.cc:3037] T f2cdbe5f94184cb085271c1114671ec4 P 1112613e1b9142908b4b0c3091e420c2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:03:29.488646 23039 raft_consensus.cc:504] T f2cdbe5f94184cb085271c1114671ec4 P 1112613e1b9142908b4b0c3091e420c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c35034b75cf40bd80f78d14645555f1" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 38395 } } peers { permanent_uuid: "6feefb0d2fee41c69f2dd1540810307e" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41559 } } peers { permanent_uuid: "1112613e1b9142908b4b0c3091e420c2" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 37463 } }
I20260926 01:03:29.488786 23039 leader_election.cc:288] T f2cdbe5f94184cb085271c1114671ec4 P 1112613e1b9142908b4b0c3091e420c2 [CANDIDATE]: Term 1 election: Requested vote from peers 0c35034b75cf40bd80f78d14645555f1 (127.3.75.1:38395), 6feefb0d2fee41c69f2dd1540810307e (127.3.75.2:41559)
I20260926 01:03:29.488910 22697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2cdbe5f94184cb085271c1114671ec4" candidate_uuid: "1112613e1b9142908b4b0c3091e420c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c35034b75cf40bd80f78d14645555f1"
I20260926 01:03:29.488946 22832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2cdbe5f94184cb085271c1114671ec4" candidate_uuid: "1112613e1b9142908b4b0c3091e420c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6feefb0d2fee41c69f2dd1540810307e"
I20260926 01:03:29.488976 22697 raft_consensus.cc:3037] T f2cdbe5f94184cb085271c1114671ec4 P 0c35034b75cf40bd80f78d14645555f1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:03:29.489010 22832 raft_consensus.cc:3037] T f2cdbe5f94184cb085271c1114671ec4 P 6feefb0d2fee41c69f2dd1540810307e [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:03:29.489768 22697 raft_consensus.cc:2454] T f2cdbe5f94184cb085271c1114671ec4 P 0c35034b75cf40bd80f78d14645555f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1112613e1b9142908b4b0c3091e420c2 in term 1.
I20260926 01:03:29.489771 22832 raft_consensus.cc:2454] T f2cdbe5f94184cb085271c1114671ec4 P 6feefb0d2fee41c69f2dd1540810307e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1112613e1b9142908b4b0c3091e420c2 in term 1.
I20260926 01:03:29.489926 22900 leader_election.cc:302] T f2cdbe5f94184cb085271c1114671ec4 P 1112613e1b9142908b4b0c3091e420c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1112613e1b9142908b4b0c3091e420c2, 6feefb0d2fee41c69f2dd1540810307e; no voters:
I20260926 01:03:29.490015 23039 raft_consensus.cc:2781] T f2cdbe5f94184cb085271c1114671ec4 P 1112613e1b9142908b4b0c3091e420c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:03:29.490141 23039 raft_consensus.cc:686] T f2cdbe5f94184cb085271c1114671ec4 P 1112613e1b9142908b4b0c3091e420c2 [term 1 LEADER]: Becoming Leader. State: Replica: 1112613e1b9142908b4b0c3091e420c2, State: Running, Role: LEADER
I20260926 01:03:29.490237 23039 consensus_queue.cc:237] T f2cdbe5f94184cb085271c1114671ec4 P 1112613e1b9142908b4b0c3091e420c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c35034b75cf40bd80f78d14645555f1" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 38395 } } peers { permanent_uuid: "6feefb0d2fee41c69f2dd1540810307e" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41559 } } peers { permanent_uuid: "1112613e1b9142908b4b0c3091e420c2" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 37463 } }
I20260926 01:03:29.490967 22551 catalog_manager.cc:5495] T f2cdbe5f94184cb085271c1114671ec4 P 1112613e1b9142908b4b0c3091e420c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1112613e1b9142908b4b0c3091e420c2 (127.3.75.3). New cstate: current_term: 1 leader_uuid: "1112613e1b9142908b4b0c3091e420c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c35034b75cf40bd80f78d14645555f1" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 38395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6feefb0d2fee41c69f2dd1540810307e" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1112613e1b9142908b4b0c3091e420c2" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 37463 } health_report { overall_health: HEALTHY } } }
I20260926 01:03:29.495301 23030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:29.493248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60292 (local address 127.3.75.3:37463)
0926 01:03:29.493287 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:29.493289 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:29.493297 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:03:29.493346 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:29.493347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:29.493360 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:29.493414 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:03:29.493495 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:29.493886 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:29.494645 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:29.494822 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:29.494866 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:29.495046 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:03:29.495120 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:03:29.495173 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:29.495210 (+ 37us) server_negotiation.cc:299] Negotiation successful
0926 01:03:29.495225 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 01:03:29.496673 22832 raft_consensus.cc:1261] T f2cdbe5f94184cb085271c1114671ec4 P 6feefb0d2fee41c69f2dd1540810307e [term 1 FOLLOWER]: Refusing update from remote peer 1112613e1b9142908b4b0c3091e420c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:03:29.496711 22697 raft_consensus.cc:1261] T f2cdbe5f94184cb085271c1114671ec4 P 0c35034b75cf40bd80f78d14645555f1 [term 1 FOLLOWER]: Refusing update from remote peer 1112613e1b9142908b4b0c3091e420c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:03:29.496999 23039 consensus_queue.cc:1035] T f2cdbe5f94184cb085271c1114671ec4 P 1112613e1b9142908b4b0c3091e420c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6feefb0d2fee41c69f2dd1540810307e" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 41559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:03:29.497131 23041 consensus_queue.cc:1035] T f2cdbe5f94184cb085271c1114671ec4 P 1112613e1b9142908b4b0c3091e420c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c35034b75cf40bd80f78d14645555f1" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 38395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:03:29.499328 23044 mvcc.cc:205] Tried to move back new op lower bound from 7333415360496414720 to 7333415360472551424. Current Snapshot: MvccSnapshot[applied={T|T < 7333415360496414720}]
I20260926 01:03:29.500406 23046 mvcc.cc:205] Tried to move back new op lower bound from 7333415360496414720 to 7333415360472551424. Current Snapshot: MvccSnapshot[applied={T|T < 7333415360496414720 or (T in {7333415360496414720})}]
I20260926 01:03:29.501363 22550 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:03:29.501454 22550 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:03:29.502067 22550 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5ab323b0faf7425aa021cb21a2883b95: Sending DeleteTablet for 3 replicas of tablet f2cdbe5f94184cb085271c1114671ec4
I20260926 01:03:29.502380 22677 tablet_service.cc:1514] Processing DeleteTablet for tablet f2cdbe5f94184cb085271c1114671ec4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:03:29 UTC) from {username='kudu', principal='kudu/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:56484
I20260926 01:03:29.502480 22946 tablet_service.cc:1514] Processing DeleteTablet for tablet f2cdbe5f94184cb085271c1114671ec4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:03:29 UTC) from {username='kudu', principal='kudu/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:60280
I20260926 01:03:29.502480 22812 tablet_service.cc:1514] Processing DeleteTablet for tablet f2cdbe5f94184cb085271c1114671ec4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:03:29 UTC) from {username='kudu', principal='kudu/127.3.75.62@KRBTEST.COM'} at 127.0.0.1:60128
I20260926 01:03:29.502578 23054 tablet_replica.cc:330] T f2cdbe5f94184cb085271c1114671ec4 P 0c35034b75cf40bd80f78d14645555f1: stopping tablet replica
I20260926 01:03:29.502660 23054 raft_consensus.cc:2229] T f2cdbe5f94184cb085271c1114671ec4 P 0c35034b75cf40bd80f78d14645555f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:03:29.502691 23055 tablet_replica.cc:330] T f2cdbe5f94184cb085271c1114671ec4 P 6feefb0d2fee41c69f2dd1540810307e: stopping tablet replica
I20260926 01:03:29.502737 23054 raft_consensus.cc:2258] T f2cdbe5f94184cb085271c1114671ec4 P 0c35034b75cf40bd80f78d14645555f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:03:29.502768 23055 raft_consensus.cc:2229] T f2cdbe5f94184cb085271c1114671ec4 P 6feefb0d2fee41c69f2dd1540810307e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:03:29.502859 23055 raft_consensus.cc:2258] T f2cdbe5f94184cb085271c1114671ec4 P 6feefb0d2fee41c69f2dd1540810307e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:03:29.503051 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 22613
I20260926 01:03:29.503162 23054 ts_tablet_manager.cc:1905] T f2cdbe5f94184cb085271c1114671ec4 P 0c35034b75cf40bd80f78d14645555f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:03:29.505651 23055 ts_tablet_manager.cc:1905] T f2cdbe5f94184cb085271c1114671ec4 P 6feefb0d2fee41c69f2dd1540810307e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:03:29.507340 23056 tablet_replica.cc:330] T f2cdbe5f94184cb085271c1114671ec4 P 1112613e1b9142908b4b0c3091e420c2: stopping tablet replica
I20260926 01:03:29.507470 23056 raft_consensus.cc:2229] T f2cdbe5f94184cb085271c1114671ec4 P 1112613e1b9142908b4b0c3091e420c2 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:03:29.507541 23055 ts_tablet_manager.cc:1918] T f2cdbe5f94184cb085271c1114671ec4 P 6feefb0d2fee41c69f2dd1540810307e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:03:29.507596 23056 raft_consensus.cc:2258] T f2cdbe5f94184cb085271c1114671ec4 P 1112613e1b9142908b4b0c3091e420c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:03:29.507650 23055 log.cc:1196] T f2cdbe5f94184cb085271c1114671ec4 P 6feefb0d2fee41c69f2dd1540810307e: Deleting WAL directory at /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/ts-1/wal/wals/f2cdbe5f94184cb085271c1114671ec4
I20260926 01:03:29.507989 23055 ts_tablet_manager.cc:1939] T f2cdbe5f94184cb085271c1114671ec4 P 6feefb0d2fee41c69f2dd1540810307e: Deleting consensus metadata
I20260926 01:03:29.508030 23056 ts_tablet_manager.cc:1905] T f2cdbe5f94184cb085271c1114671ec4 P 1112613e1b9142908b4b0c3091e420c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:03:29.508425 22536 catalog_manager.cc:4841] TS 6feefb0d2fee41c69f2dd1540810307e (127.3.75.2:41559): tablet f2cdbe5f94184cb085271c1114671ec4 (table test-table [id=c07b65ab604244deba7e5b24a287dd3c]) successfully deleted
W20260926 01:03:29.508667 22537 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.75.1:38395 (error 108)
I20260926 01:03:29.508926 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 22748
I20260926 01:03:29.509548 23056 ts_tablet_manager.cc:1918] T f2cdbe5f94184cb085271c1114671ec4 P 1112613e1b9142908b4b0c3091e420c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:03:29.509665 23056 log.cc:1196] T f2cdbe5f94184cb085271c1114671ec4 P 1112613e1b9142908b4b0c3091e420c2: Deleting WAL directory at /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384372073861-3372-0/minicluster-data/ts-2/wal/wals/f2cdbe5f94184cb085271c1114671ec4
I20260926 01:03:29.509944 23056 ts_tablet_manager.cc:1939] T f2cdbe5f94184cb085271c1114671ec4 P 1112613e1b9142908b4b0c3091e420c2: Deleting consensus metadata
I20260926 01:03:29.510310 22536 catalog_manager.cc:4841] TS 1112613e1b9142908b4b0c3091e420c2 (127.3.75.3:37463): tablet f2cdbe5f94184cb085271c1114671ec4 (table test-table [id=c07b65ab604244deba7e5b24a287dd3c]) successfully deleted
I20260926 01:03:29.511329 23025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:29.511232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.1:38395 (local address 127.0.0.1:56498)
0926 01:03:29.511269 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:29.511306 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.1:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260926 01:03:29.511469 22537 catalog_manager.cc:4584] TS 0c35034b75cf40bd80f78d14645555f1 (127.3.75.1:38395): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f2cdbe5f94184cb085271c1114671ec4: Network error: Client connection negotiation failed: client connection to 127.3.75.1:38395: connect: Connection refused (error 111)
I20260926 01:03:29.515110 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 22883
I20260926 01:03:29.520092 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 22519
2026-09-26T01:03:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1893 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-26T01:03:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:03:29Z Disabled control of system clock
I20260926 01:03:29.547705 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:42385
--webserver_interface=127.3.75.62
--webserver_port=0
--builtin_ntp_servers=127.3.75.20:38239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.75.62:42385
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 01:03:29.626593 23061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:03:29.626838 23061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:03:29.626858 23061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:03:29.628087 23061 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:03:29.628129 23061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:03:29.628141 23061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:03:29.628153 23061 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:03:29.628166 23061 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:03:29.629484 23061 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:38239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.75.62:42385
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:42385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:03:29.629735 23061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:03:29.630016 23061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:03:29.634627 23069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:29.634725 23067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:29.634650 23066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:03:29.634898 23061 server_base.cc:1034] running on GCE node
I20260926 01:03:29.635142 23061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:03:29.635476 23061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:03:29.636646 23061 hybrid_clock.cc:648] HybridClock initialized: now 1790384609636619 us; error 41 us; skew 500 ppm
I20260926 01:03:29.637871 23061 webserver.cc:460] Webserver started at http://127.3.75.62:36517/ using document root <none> and password file <none>
I20260926 01:03:29.638135 23061 fs_manager.cc:359] Metadata directory not provided
I20260926 01:03:29.638216 23061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:03:29.638368 23061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:03:29.639899 23061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/master-0/data/instance:
uuid: "ccc552a5d1ec40e69ce6183b57a8a2e8"
format_stamp: "Formatted at 2026-09-26 01:03:29 on dist-test-slave-7v10"
server_key: "5557d00b0152004f3b50237f2945e809"
server_key_iv: "caf908308953c70f90bfe3ec466e21e7"
server_key_version: "encryptionkey@0"
I20260926 01:03:29.640358 23061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/master-0/wal/instance:
uuid: "ccc552a5d1ec40e69ce6183b57a8a2e8"
format_stamp: "Formatted at 2026-09-26 01:03:29 on dist-test-slave-7v10"
server_key: "5557d00b0152004f3b50237f2945e809"
server_key_iv: "caf908308953c70f90bfe3ec466e21e7"
server_key_version: "encryptionkey@0"
I20260926 01:03:29.642290 23061 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 01:03:29.643430 23075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:29.643704 23061 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 01:03:29.643802 23061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/master-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/master-0/wal
uuid: "ccc552a5d1ec40e69ce6183b57a8a2e8"
format_stamp: "Formatted at 2026-09-26 01:03:29 on dist-test-slave-7v10"
server_key: "5557d00b0152004f3b50237f2945e809"
server_key_iv: "caf908308953c70f90bfe3ec466e21e7"
server_key_version: "encryptionkey@0"
I20260926 01:03:29.643890 23061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:03:29.671725 23061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:03:29.672071 23061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:03:29.672245 23061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:03:29.676190 23061 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.62:42385
I20260926 01:03:29.676245 23128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.62:42385 every 8 connection(s)
I20260926 01:03:29.676614 23061 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
I20260926 01:03:29.677584 23129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:03:29.679847 23129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccc552a5d1ec40e69ce6183b57a8a2e8: Bootstrap starting.
I20260926 01:03:29.680383 23129 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccc552a5d1ec40e69ce6183b57a8a2e8: Neither blocks nor log segments found. Creating new log.
I20260926 01:03:29.680620 23129 log.cc:824] T 00000000000000000000000000000000 P ccc552a5d1ec40e69ce6183b57a8a2e8: Log is configured to *not* fsync() on all Append() calls
I20260926 01:03:29.681262 23129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccc552a5d1ec40e69ce6183b57a8a2e8: No bootstrap required, opened a new log
I20260926 01:03:29.682165 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 23061
I20260926 01:03:29.682276 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/master-0/wal/instance
I20260926 01:03:29.682473 3372 external_mini_cluster.cc:1428] Setting key 7f7dfa212b782a65117a0955036fc223
I20260926 01:03:29.683424 23129 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccc552a5d1ec40e69ce6183b57a8a2e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccc552a5d1ec40e69ce6183b57a8a2e8" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42385 } }
I20260926 01:03:29.683599 23129 raft_consensus.cc:374] T 00000000000000000000000000000000 P ccc552a5d1ec40e69ce6183b57a8a2e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:03:29.683645 23129 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccc552a5d1ec40e69ce6183b57a8a2e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccc552a5d1ec40e69ce6183b57a8a2e8, State: Initialized, Role: FOLLOWER
I20260926 01:03:29.683816 23129 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccc552a5d1ec40e69ce6183b57a8a2e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccc552a5d1ec40e69ce6183b57a8a2e8" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42385 } }
I20260926 01:03:29.683909 23129 raft_consensus.cc:388] T 00000000000000000000000000000000 P ccc552a5d1ec40e69ce6183b57a8a2e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:03:29.683947 23129 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccc552a5d1ec40e69ce6183b57a8a2e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:03:29.683997 23129 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccc552a5d1ec40e69ce6183b57a8a2e8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:03:29.684532 23129 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccc552a5d1ec40e69ce6183b57a8a2e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccc552a5d1ec40e69ce6183b57a8a2e8" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42385 } }
I20260926 01:03:29.684672 23129 leader_election.cc:302] T 00000000000000000000000000000000 P ccc552a5d1ec40e69ce6183b57a8a2e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ccc552a5d1ec40e69ce6183b57a8a2e8; no voters:
I20260926 01:03:29.684877 23129 leader_election.cc:288] T 00000000000000000000000000000000 P ccc552a5d1ec40e69ce6183b57a8a2e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:03:29.684908 23134 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccc552a5d1ec40e69ce6183b57a8a2e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:03:29.685015 23134 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccc552a5d1ec40e69ce6183b57a8a2e8 [term 1 LEADER]: Becoming Leader. State: Replica: ccc552a5d1ec40e69ce6183b57a8a2e8, State: Running, Role: LEADER
I20260926 01:03:29.685168 23129 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccc552a5d1ec40e69ce6183b57a8a2e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:03:29.685159 23134 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccc552a5d1ec40e69ce6183b57a8a2e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccc552a5d1ec40e69ce6183b57a8a2e8" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42385 } }
I20260926 01:03:29.685724 23136 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccc552a5d1ec40e69ce6183b57a8a2e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ccc552a5d1ec40e69ce6183b57a8a2e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccc552a5d1ec40e69ce6183b57a8a2e8" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42385 } } }
I20260926 01:03:29.685904 23136 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccc552a5d1ec40e69ce6183b57a8a2e8 [sys.catalog]: This master's current role is: LEADER
I20260926 01:03:29.686019 23135 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccc552a5d1ec40e69ce6183b57a8a2e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccc552a5d1ec40e69ce6183b57a8a2e8. Latest consensus state: current_term: 1 leader_uuid: "ccc552a5d1ec40e69ce6183b57a8a2e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccc552a5d1ec40e69ce6183b57a8a2e8" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 42385 } } }
I20260926 01:03:29.686420 23146 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:03:29.686445 23135 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccc552a5d1ec40e69ce6183b57a8a2e8 [sys.catalog]: This master's current role is: LEADER
I20260926 01:03:29.687888 23146 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:03:29.688733 23133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:29.683113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47824 (local address 127.3.75.62:42385)
0926 01:03:29.683372 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:29.683380 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:29.683405 (+ 25us) server_negotiation.cc:407] Connection header received
0926 01:03:29.683498 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:29.683504 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:29.683557 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:29.683710 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:29.683996 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:29.685482 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:29.686612 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:29.687158 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:29.687305 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:29.687962 (+ 657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:29.687966 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:29.687967 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:03:29.688181 (+ 214us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:29.688223 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:03:29.688237 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:29.688310 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:29.688444 (+ 134us) server_negotiation.cc:299] Negotiation successful
0926 01:03:29.688525 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":103,"threads_started":1}
I20260926 01:03:29.689872 23146 catalog_manager.cc:1347] Generated new cluster ID: f202edfa7c114dbcba60b2705b9cf9f8
I20260926 01:03:29.689954 23146 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:03:29.707211 23146 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:03:29.707934 23146 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:03:29.711540 23146 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ccc552a5d1ec40e69ce6183b57a8a2e8: Generated new TSK 0
I20260926 01:03:29.711768 23146 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:03:29.719789 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.1:0
--local_ip_for_outbound_sockets=127.3.75.1
--webserver_interface=127.3.75.1
--webserver_port=0
--tserver_master_addrs=127.3.75.62:42385
--builtin_ntp_servers=127.3.75.20:38239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 01:03:29.802395 23153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:03:29.802610 23153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:03:29.802646 23153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:03:29.803795 23153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:03:29.803858 23153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:03:29.803895 23153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.1
I20260926 01:03:29.805241 23153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:38239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.75.1
--webserver_port=0
--tserver_master_addrs=127.3.75.62:42385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.1
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:03:29.805491 23153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:03:29.805776 23153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:03:29.811430 23161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:29.811403 23158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:29.811537 23159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:03:29.811777 23153 server_base.cc:1034] running on GCE node
I20260926 01:03:29.811997 23153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:03:29.812192 23153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:03:29.813328 23153 hybrid_clock.cc:648] HybridClock initialized: now 1790384609813315 us; error 27 us; skew 500 ppm
I20260926 01:03:29.814556 23153 webserver.cc:460] Webserver started at http://127.3.75.1:35699/ using document root <none> and password file <none>
I20260926 01:03:29.814742 23153 fs_manager.cc:359] Metadata directory not provided
I20260926 01:03:29.814781 23153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:03:29.814865 23153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:03:29.815943 23153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/ts-0/data/instance:
uuid: "cc115eb90df347eeb71d0cd9b3e01afb"
format_stamp: "Formatted at 2026-09-26 01:03:29 on dist-test-slave-7v10"
server_key: "ad83e8f81898aa4cf5b67563bcd73038"
server_key_iv: "7fd7c76e876d4e5f14217a903f6654ee"
server_key_version: "encryptionkey@0"
I20260926 01:03:29.816241 23153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/ts-0/wal/instance:
uuid: "cc115eb90df347eeb71d0cd9b3e01afb"
format_stamp: "Formatted at 2026-09-26 01:03:29 on dist-test-slave-7v10"
server_key: "ad83e8f81898aa4cf5b67563bcd73038"
server_key_iv: "7fd7c76e876d4e5f14217a903f6654ee"
server_key_version: "encryptionkey@0"
I20260926 01:03:29.817528 23153 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 01:03:29.818140 23167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:29.818303 23153 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 01:03:29.818370 23153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/ts-0/wal
uuid: "cc115eb90df347eeb71d0cd9b3e01afb"
format_stamp: "Formatted at 2026-09-26 01:03:29 on dist-test-slave-7v10"
server_key: "ad83e8f81898aa4cf5b67563bcd73038"
server_key_iv: "7fd7c76e876d4e5f14217a903f6654ee"
server_key_version: "encryptionkey@0"
I20260926 01:03:29.818421 23153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:03:29.830370 23153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:03:29.830600 23153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:03:29.830696 23153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:03:29.830865 23153 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:03:29.831278 23153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:03:29.831327 23153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:29.831357 23153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:03:29.831369 23153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:29.837723 23153 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.1:35541
I20260926 01:03:29.837771 23281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.1:35541 every 8 connection(s)
I20260926 01:03:29.838130 23153 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/ts-0/data/info.pb
I20260926 01:03:29.843550 23133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:29.838965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.1:47249 (local address 127.3.75.62:42385)
0926 01:03:29.839043 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:29.839045 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:29.839316 (+ 271us) server_negotiation.cc:407] Connection header received
0926 01:03:29.839937 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:29.839938 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:29.839951 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:29.840003 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:29.840701 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:29.841188 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:29.842249 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:29.842428 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:29.842477 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:29.843341 (+ 864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:29.843342 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:29.843342 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:03:29.843346 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:29.843359 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:03:29.843363 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:29.843405 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:29.843467 (+ 62us) server_negotiation.cc:299] Negotiation successful
0926 01:03:29.843492 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 01:03:29.843768 23284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:29.838948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:42385 (local address 127.3.75.1:47249)
0926 01:03:29.839169 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:29.839205 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:29.839782 (+ 577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:29.840065 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:29.840069 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:29.840077 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:03:29.840621 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:29.840625 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:29.841366 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:29.841367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:29.842169 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:29.842172 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:29.842501 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:29.842501 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:29.842608 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:29.843206 (+ 598us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 01:03:29.843207 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:03:29.843240 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 01:03:29.843264 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:03:29.843405 (+ 141us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:03:29.843408 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:03:29.843410 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 01:03:29.843456 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 01:03:29.843530 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":88,"threads_started":1}
I20260926 01:03:29.844332 23282 heartbeater.cc:347] Connected to a master server at 127.3.75.62:42385
I20260926 01:03:29.844426 23282 heartbeater.cc:464] Registering TS with master...
I20260926 01:03:29.844609 23282 heartbeater.cc:511] Master 127.3.75.62:42385 requested a full tablet report, sending...
I20260926 01:03:29.844597 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 23153
I20260926 01:03:29.844732 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/ts-0/wal/instance
I20260926 01:03:29.844930 3372 external_mini_cluster.cc:1428] Setting key 87a9c2d232b28066df9c5f4996fd1a12
I20260926 01:03:29.845171 23092 ts_manager.cc:194] Registered new tserver with Master: cc115eb90df347eeb71d0cd9b3e01afb (127.3.75.1:35541)
I20260926 01:03:29.846040 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.2:0
--local_ip_for_outbound_sockets=127.3.75.2
--webserver_interface=127.3.75.2
--webserver_port=0
--tserver_master_addrs=127.3.75.62:42385
--builtin_ntp_servers=127.3.75.20:38239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 01:03:29.846050 23092 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.75.1:47249
W20260926 01:03:29.926685 23285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:03:29.926935 23285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:03:29.926985 23285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:03:29.928108 23285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:03:29.928188 23285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:03:29.928232 23285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.2
I20260926 01:03:29.929608 23285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:38239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.75.2
--webserver_port=0
--tserver_master_addrs=127.3.75.62:42385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.2
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:03:29.929904 23285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:03:29.930213 23285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:03:29.935266 23293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:29.935257 23291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:29.935366 23290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:03:29.935616 23285 server_base.cc:1034] running on GCE node
I20260926 01:03:29.935819 23285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:03:29.936051 23285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:03:29.937234 23285 hybrid_clock.cc:648] HybridClock initialized: now 1790384609937214 us; error 43 us; skew 500 ppm
I20260926 01:03:29.938411 23285 webserver.cc:460] Webserver started at http://127.3.75.2:35103/ using document root <none> and password file <none>
I20260926 01:03:29.938671 23285 fs_manager.cc:359] Metadata directory not provided
I20260926 01:03:29.938750 23285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:03:29.938921 23285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:03:29.939985 23285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/ts-1/data/instance:
uuid: "05eb1d0f7a384373b74849d0496fed00"
format_stamp: "Formatted at 2026-09-26 01:03:29 on dist-test-slave-7v10"
server_key: "57acf52928b618ab7f909f91a67dc79b"
server_key_iv: "573bb92e4c62f09d4eef327bac6610ad"
server_key_version: "encryptionkey@0"
I20260926 01:03:29.940352 23285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/ts-1/wal/instance:
uuid: "05eb1d0f7a384373b74849d0496fed00"
format_stamp: "Formatted at 2026-09-26 01:03:29 on dist-test-slave-7v10"
server_key: "57acf52928b618ab7f909f91a67dc79b"
server_key_iv: "573bb92e4c62f09d4eef327bac6610ad"
server_key_version: "encryptionkey@0"
I20260926 01:03:29.941633 23285 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 01:03:29.942299 23299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:29.942466 23285 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 01:03:29.942560 23285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/ts-1/wal
uuid: "05eb1d0f7a384373b74849d0496fed00"
format_stamp: "Formatted at 2026-09-26 01:03:29 on dist-test-slave-7v10"
server_key: "57acf52928b618ab7f909f91a67dc79b"
server_key_iv: "573bb92e4c62f09d4eef327bac6610ad"
server_key_version: "encryptionkey@0"
I20260926 01:03:29.942642 23285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:03:29.959123 23285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:03:29.959407 23285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:03:29.959555 23285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:03:29.959769 23285 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:03:29.960083 23285 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:03:29.960139 23285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:29.960189 23285 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:03:29.960216 23285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:29.965792 23285 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.2:32813
I20260926 01:03:29.965838 23413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.2:32813 every 8 connection(s)
I20260926 01:03:29.966148 23285 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/ts-1/data/info.pb
I20260926 01:03:29.970968 23133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:29.966892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.2:48139 (local address 127.3.75.62:42385)
0926 01:03:29.966983 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:29.966985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:29.967160 (+ 175us) server_negotiation.cc:407] Connection header received
0926 01:03:29.967623 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:29.967624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:29.967633 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:29.967693 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:29.968312 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:29.968786 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:29.969695 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:29.969871 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:29.969920 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:29.970705 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:29.970706 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:29.970707 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:03:29.970710 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:29.970720 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:03:29.970723 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:29.970761 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:29.970845 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 01:03:29.970863 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 01:03:29.971151 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 23285
I20260926 01:03:29.971256 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/ts-1/wal/instance
I20260926 01:03:29.971305 23416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:29.966829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:42385 (local address 127.3.75.2:48139)
0926 01:03:29.967040 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:29.967076 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:29.967543 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:29.967761 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:29.967765 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:29.967772 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:03:29.968244 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:29.968248 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:29.968869 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:29.968869 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:29.969621 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:29.969622 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:29.969947 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:29.969948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:29.970048 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:29.970584 (+ 536us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 01:03:29.970585 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:03:29.970611 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 01:03:29.970632 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:03:29.970778 (+ 146us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:03:29.970782 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:03:29.970784 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 01:03:29.970835 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 01:03:29.970999 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":97,"threads_started":1}
I20260926 01:03:29.971474 3372 external_mini_cluster.cc:1428] Setting key 7d86df03029c328155bab5bb8c57edb1
I20260926 01:03:29.971740 23414 heartbeater.cc:347] Connected to a master server at 127.3.75.62:42385
I20260926 01:03:29.971823 23414 heartbeater.cc:464] Registering TS with master...
I20260926 01:03:29.971992 23414 heartbeater.cc:511] Master 127.3.75.62:42385 requested a full tablet report, sending...
I20260926 01:03:29.972383 23092 ts_manager.cc:194] Registered new tserver with Master: 05eb1d0f7a384373b74849d0496fed00 (127.3.75.2:32813)
I20260926 01:03:29.972491 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.75.3:0
--local_ip_for_outbound_sockets=127.3.75.3
--webserver_interface=127.3.75.3
--webserver_port=0
--tserver_master_addrs=127.3.75.62:42385
--builtin_ntp_servers=127.3.75.20:38239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 01:03:29.972823 23092 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.75.2:48139
W20260926 01:03:30.053063 23417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:03:30.053305 23417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:03:30.053350 23417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:03:30.054490 23417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:03:30.054605 23417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:03:30.054651 23417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.75.3
I20260926 01:03:30.056114 23417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:38239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.75.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.75.3
--webserver_port=0
--tserver_master_addrs=127.3.75.62:42385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.75.3
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:03:30.056383 23417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:03:30.056689 23417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:03:30.061862 23422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:30.061856 23423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:03:30.061902 23417 server_base.cc:1034] running on GCE node
W20260926 01:03:30.061856 23425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:03:30.062249 23417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:03:30.062495 23417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:03:30.063658 23417 hybrid_clock.cc:648] HybridClock initialized: now 1790384610063643 us; error 51 us; skew 500 ppm
I20260926 01:03:30.064867 23417 webserver.cc:460] Webserver started at http://127.3.75.3:43955/ using document root <none> and password file <none>
I20260926 01:03:30.065156 23417 fs_manager.cc:359] Metadata directory not provided
I20260926 01:03:30.065230 23417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:03:30.065378 23417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:03:30.066424 23417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/ts-2/data/instance:
uuid: "2d88144e1c0548c298e7dd18973eea78"
format_stamp: "Formatted at 2026-09-26 01:03:30 on dist-test-slave-7v10"
server_key: "9eecb2915eec1339bede13deef275388"
server_key_iv: "ff80e52fe95dd65184b0bccca26e6817"
server_key_version: "encryptionkey@0"
I20260926 01:03:30.066794 23417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/ts-2/wal/instance:
uuid: "2d88144e1c0548c298e7dd18973eea78"
format_stamp: "Formatted at 2026-09-26 01:03:30 on dist-test-slave-7v10"
server_key: "9eecb2915eec1339bede13deef275388"
server_key_iv: "ff80e52fe95dd65184b0bccca26e6817"
server_key_version: "encryptionkey@0"
I20260926 01:03:30.068048 23417 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:03:30.068729 23431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:30.068893 23417 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 01:03:30.069005 23417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/ts-2/wal
uuid: "2d88144e1c0548c298e7dd18973eea78"
format_stamp: "Formatted at 2026-09-26 01:03:30 on dist-test-slave-7v10"
server_key: "9eecb2915eec1339bede13deef275388"
server_key_iv: "ff80e52fe95dd65184b0bccca26e6817"
server_key_version: "encryptionkey@0"
I20260926 01:03:30.069096 23417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:03:30.083329 23417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:03:30.083634 23417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:03:30.083788 23417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:03:30.084021 23417 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:03:30.084369 23417 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:03:30.084427 23417 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:30.084470 23417 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:03:30.084497 23417 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:30.090615 23417 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.3:34807
I20260926 01:03:30.090713 23545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.3:34807 every 8 connection(s)
I20260926 01:03:30.091075 23417 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/ts-2/data/info.pb
I20260926 01:03:30.096084 23133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:30.091915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.3:41117 (local address 127.3.75.62:42385)
0926 01:03:30.092007 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:30.092009 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:30.092220 (+ 211us) server_negotiation.cc:407] Connection header received
0926 01:03:30.092690 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:30.092691 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:30.092701 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:30.092762 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:30.093385 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:30.093869 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:30.094844 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:30.095093 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:30.095145 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:30.095868 (+ 723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:30.095869 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:30.095869 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:03:30.095874 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:30.095887 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:03:30.095891 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:30.095937 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:30.096004 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 01:03:30.096027 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 01:03:30.096297 23548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:30.091893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:42385 (local address 127.3.75.3:41117)
0926 01:03:30.092085 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:30.092127 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:30.092608 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:30.092812 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:30.092816 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:30.092823 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:03:30.093313 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:30.093317 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:30.093986 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:30.093987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:30.094766 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:30.094768 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:30.095167 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:30.095168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:30.095276 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:30.095738 (+ 462us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 01:03:30.095739 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:03:30.095766 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 01:03:30.095788 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:03:30.095939 (+ 151us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:03:30.095941 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:03:30.095944 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:03:30.095989 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 01:03:30.096062 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":92,"threads_started":1}
I20260926 01:03:30.096817 23546 heartbeater.cc:347] Connected to a master server at 127.3.75.62:42385
I20260926 01:03:30.096925 23546 heartbeater.cc:464] Registering TS with master...
I20260926 01:03:30.097151 23546 heartbeater.cc:511] Master 127.3.75.62:42385 requested a full tablet report, sending...
I20260926 01:03:30.097203 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 23417
I20260926 01:03:30.097316 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/ts-2/wal/instance
I20260926 01:03:30.097523 3372 external_mini_cluster.cc:1428] Setting key b4c698bb74c6391394f439f4c50d79a2
I20260926 01:03:30.097580 23092 ts_manager.cc:194] Registered new tserver with Master: 2d88144e1c0548c298e7dd18973eea78 (127.3.75.3:34807)
I20260926 01:03:30.098071 23092 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.75.3:41117
I20260926 01:03:30.098403 3372 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 01:03:30.102394 23133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:30.099809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47838 (local address 127.3.75.62:42385)
0926 01:03:30.099869 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:30.099871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:30.099960 (+ 89us) server_negotiation.cc:407] Connection header received
0926 01:03:30.100030 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:30.100031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:30.100040 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:30.100091 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:30.100232 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:30.100663 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:30.101352 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:30.101558 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:30.101611 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:30.102183 (+ 572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:30.102185 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:30.102185 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:03:30.102188 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:30.102199 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:03:30.102203 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:30.102239 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:30.102327 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 01:03:30.102346 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 01:03:30.103909 23091 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47838:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:03:30.105294 23091 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:03:30.111833 23556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:30.107660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49618 (local address 127.3.75.2:32813)
0926 01:03:30.107854 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:30.107859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:30.108071 (+ 212us) server_negotiation.cc:407] Connection header received
0926 01:03:30.108195 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:30.108198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:30.108215 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:30.108289 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:03:30.108476 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:30.109504 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:30.110758 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:30.111445 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:30.111508 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:30.111578 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:30.111709 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 01:03:30.111751 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":85,"threads_started":1}
I20260926 01:03:30.111837 23555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:30.107762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.1:35541 (local address 127.0.0.1:56804)
0926 01:03:30.108202 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:30.108210 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:30.108251 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:30.108514 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:30.108515 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:30.108517 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:03:30.108558 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:30.108559 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:30.109413 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:30.109414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:30.110502 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:30.110505 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:30.111464 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:30.111465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:30.111580 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:30.111595 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 01:03:30.111677 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 01:03:30.111743 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"mutex_wait_us":77,"thread_start_us":65,"threads_started":1}
I20260926 01:03:30.112082 23559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:30.107950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56804 (local address 127.3.75.1:35541)
0926 01:03:30.108132 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:30.108137 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:30.108311 (+ 174us) server_negotiation.cc:407] Connection header received
0926 01:03:30.108355 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:30.108358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:30.108374 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:30.108443 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:03:30.108642 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:30.109321 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:30.110594 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:30.111364 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:30.111443 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:30.111541 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:30.111959 (+ 418us) server_negotiation.cc:299] Negotiation successful
0926 01:03:30.112005 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":81,"threads_started":1}
I20260926 01:03:30.111837 23558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:30.107536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.2:32813 (local address 127.0.0.1:49618)
0926 01:03:30.107980 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:30.107991 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:30.108074 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:30.108360 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:30.108361 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:30.108363 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:03:30.108394 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:30.108396 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:30.109604 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:30.109604 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:30.110673 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:30.110674 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:30.111541 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:30.111542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:30.111591 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:30.111596 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 01:03:30.111688 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 01:03:30.111743 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"mutex_wait_us":27,"thread_start_us":73,"threads_started":1}
I20260926 01:03:30.112768 23554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:30.107536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.3:34807 (local address 127.0.0.1:34500)
0926 01:03:30.107713 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:30.107749 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:30.107809 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:30.108120 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:30.108123 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:30.108126 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:03:30.108186 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:30.108188 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:30.109457 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:30.109458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:30.110672 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:30.110674 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:30.112509 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:30.112510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:30.112575 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:30.112578 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:03:30.112662 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 01:03:30.112702 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":29,"thread_start_us":76,"threads_started":1}
I20260926 01:03:30.112794 23557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:30.107724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34500 (local address 127.3.75.3:34807)
0926 01:03:30.107930 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:30.107933 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:30.107941 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:03:30.107969 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:30.107971 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:30.107982 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:30.108084 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:03:30.108275 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:30.109366 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:30.111612 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:30.112360 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:30.112430 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:30.112514 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:30.112685 (+ 171us) server_negotiation.cc:299] Negotiation successful
0926 01:03:30.112720 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":79,"threads_started":1}
I20260926 01:03:30.112772 23216 tablet_service.cc:1467] Processing CreateTablet for tablet 21202d9bd3c34f6b8cbb1d372727071e (DEFAULT_TABLE table=test-table [id=6728aa05763f458fa054d4b53109f90b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:03:30.112890 23348 tablet_service.cc:1467] Processing CreateTablet for tablet 21202d9bd3c34f6b8cbb1d372727071e (DEFAULT_TABLE table=test-table [id=6728aa05763f458fa054d4b53109f90b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:03:30.113154 23348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21202d9bd3c34f6b8cbb1d372727071e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:03:30.113147 23216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21202d9bd3c34f6b8cbb1d372727071e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:03:30.115708 23480 tablet_service.cc:1467] Processing CreateTablet for tablet 21202d9bd3c34f6b8cbb1d372727071e (DEFAULT_TABLE table=test-table [id=6728aa05763f458fa054d4b53109f90b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:03:30.116056 23480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21202d9bd3c34f6b8cbb1d372727071e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:03:30.116523 23560 tablet_bootstrap.cc:492] T 21202d9bd3c34f6b8cbb1d372727071e P 05eb1d0f7a384373b74849d0496fed00: Bootstrap starting.
I20260926 01:03:30.117249 23560 tablet_bootstrap.cc:654] T 21202d9bd3c34f6b8cbb1d372727071e P 05eb1d0f7a384373b74849d0496fed00: Neither blocks nor log segments found. Creating new log.
I20260926 01:03:30.117604 23560 log.cc:824] T 21202d9bd3c34f6b8cbb1d372727071e P 05eb1d0f7a384373b74849d0496fed00: Log is configured to *not* fsync() on all Append() calls
I20260926 01:03:30.117972 23561 tablet_bootstrap.cc:492] T 21202d9bd3c34f6b8cbb1d372727071e P cc115eb90df347eeb71d0cd9b3e01afb: Bootstrap starting.
I20260926 01:03:30.118602 23561 tablet_bootstrap.cc:654] T 21202d9bd3c34f6b8cbb1d372727071e P cc115eb90df347eeb71d0cd9b3e01afb: Neither blocks nor log segments found. Creating new log.
I20260926 01:03:30.118947 23561 log.cc:824] T 21202d9bd3c34f6b8cbb1d372727071e P cc115eb90df347eeb71d0cd9b3e01afb: Log is configured to *not* fsync() on all Append() calls
I20260926 01:03:30.119514 23560 tablet_bootstrap.cc:492] T 21202d9bd3c34f6b8cbb1d372727071e P 05eb1d0f7a384373b74849d0496fed00: No bootstrap required, opened a new log
I20260926 01:03:30.119673 23564 tablet_bootstrap.cc:492] T 21202d9bd3c34f6b8cbb1d372727071e P 2d88144e1c0548c298e7dd18973eea78: Bootstrap starting.
I20260926 01:03:30.119588 23560 ts_tablet_manager.cc:1397] T 21202d9bd3c34f6b8cbb1d372727071e P 05eb1d0f7a384373b74849d0496fed00: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 01:03:30.119786 23561 tablet_bootstrap.cc:492] T 21202d9bd3c34f6b8cbb1d372727071e P cc115eb90df347eeb71d0cd9b3e01afb: No bootstrap required, opened a new log
I20260926 01:03:30.119846 23561 ts_tablet_manager.cc:1397] T 21202d9bd3c34f6b8cbb1d372727071e P cc115eb90df347eeb71d0cd9b3e01afb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:03:30.120347 23564 tablet_bootstrap.cc:654] T 21202d9bd3c34f6b8cbb1d372727071e P 2d88144e1c0548c298e7dd18973eea78: Neither blocks nor log segments found. Creating new log.
I20260926 01:03:30.120715 23564 log.cc:824] T 21202d9bd3c34f6b8cbb1d372727071e P 2d88144e1c0548c298e7dd18973eea78: Log is configured to *not* fsync() on all Append() calls
I20260926 01:03:30.121428 23564 tablet_bootstrap.cc:492] T 21202d9bd3c34f6b8cbb1d372727071e P 2d88144e1c0548c298e7dd18973eea78: No bootstrap required, opened a new log
I20260926 01:03:30.121483 23564 ts_tablet_manager.cc:1397] T 21202d9bd3c34f6b8cbb1d372727071e P 2d88144e1c0548c298e7dd18973eea78: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 01:03:30.122426 23560 raft_consensus.cc:348] T 21202d9bd3c34f6b8cbb1d372727071e P 05eb1d0f7a384373b74849d0496fed00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d88144e1c0548c298e7dd18973eea78" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 34807 } } peers { permanent_uuid: "05eb1d0f7a384373b74849d0496fed00" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 32813 } } peers { permanent_uuid: "cc115eb90df347eeb71d0cd9b3e01afb" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 35541 } }
I20260926 01:03:30.122601 23560 raft_consensus.cc:374] T 21202d9bd3c34f6b8cbb1d372727071e P 05eb1d0f7a384373b74849d0496fed00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:03:30.122640 23560 raft_consensus.cc:729] T 21202d9bd3c34f6b8cbb1d372727071e P 05eb1d0f7a384373b74849d0496fed00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05eb1d0f7a384373b74849d0496fed00, State: Initialized, Role: FOLLOWER
I20260926 01:03:30.122701 23561 raft_consensus.cc:348] T 21202d9bd3c34f6b8cbb1d372727071e P cc115eb90df347eeb71d0cd9b3e01afb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d88144e1c0548c298e7dd18973eea78" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 34807 } } peers { permanent_uuid: "05eb1d0f7a384373b74849d0496fed00" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 32813 } } peers { permanent_uuid: "cc115eb90df347eeb71d0cd9b3e01afb" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 35541 } }
I20260926 01:03:30.122788 23560 consensus_queue.cc:260] T 21202d9bd3c34f6b8cbb1d372727071e P 05eb1d0f7a384373b74849d0496fed00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d88144e1c0548c298e7dd18973eea78" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 34807 } } peers { permanent_uuid: "05eb1d0f7a384373b74849d0496fed00" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 32813 } } peers { permanent_uuid: "cc115eb90df347eeb71d0cd9b3e01afb" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 35541 } }
I20260926 01:03:30.122864 23561 raft_consensus.cc:374] T 21202d9bd3c34f6b8cbb1d372727071e P cc115eb90df347eeb71d0cd9b3e01afb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:03:30.122941 23561 raft_consensus.cc:729] T 21202d9bd3c34f6b8cbb1d372727071e P cc115eb90df347eeb71d0cd9b3e01afb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc115eb90df347eeb71d0cd9b3e01afb, State: Initialized, Role: FOLLOWER
I20260926 01:03:30.123039 23561 consensus_queue.cc:260] T 21202d9bd3c34f6b8cbb1d372727071e P cc115eb90df347eeb71d0cd9b3e01afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d88144e1c0548c298e7dd18973eea78" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 34807 } } peers { permanent_uuid: "05eb1d0f7a384373b74849d0496fed00" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 32813 } } peers { permanent_uuid: "cc115eb90df347eeb71d0cd9b3e01afb" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 35541 } }
I20260926 01:03:30.123098 23560 ts_tablet_manager.cc:1428] T 21202d9bd3c34f6b8cbb1d372727071e P 05eb1d0f7a384373b74849d0496fed00: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 01:03:30.123262 23414 heartbeater.cc:503] Master 127.3.75.62:42385 was elected leader, sending a full tablet report...
I20260926 01:03:30.123299 23561 ts_tablet_manager.cc:1428] T 21202d9bd3c34f6b8cbb1d372727071e P cc115eb90df347eeb71d0cd9b3e01afb: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260926 01:03:30.123436 23282 heartbeater.cc:503] Master 127.3.75.62:42385 was elected leader, sending a full tablet report...
I20260926 01:03:30.124066 23564 raft_consensus.cc:348] T 21202d9bd3c34f6b8cbb1d372727071e P 2d88144e1c0548c298e7dd18973eea78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d88144e1c0548c298e7dd18973eea78" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 34807 } } peers { permanent_uuid: "05eb1d0f7a384373b74849d0496fed00" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 32813 } } peers { permanent_uuid: "cc115eb90df347eeb71d0cd9b3e01afb" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 35541 } }
I20260926 01:03:30.124248 23564 raft_consensus.cc:374] T 21202d9bd3c34f6b8cbb1d372727071e P 2d88144e1c0548c298e7dd18973eea78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:03:30.124300 23564 raft_consensus.cc:729] T 21202d9bd3c34f6b8cbb1d372727071e P 2d88144e1c0548c298e7dd18973eea78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d88144e1c0548c298e7dd18973eea78, State: Initialized, Role: FOLLOWER
I20260926 01:03:30.124501 23564 consensus_queue.cc:260] T 21202d9bd3c34f6b8cbb1d372727071e P 2d88144e1c0548c298e7dd18973eea78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d88144e1c0548c298e7dd18973eea78" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 34807 } } peers { permanent_uuid: "05eb1d0f7a384373b74849d0496fed00" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 32813 } } peers { permanent_uuid: "cc115eb90df347eeb71d0cd9b3e01afb" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 35541 } }
I20260926 01:03:30.124759 23564 ts_tablet_manager.cc:1428] T 21202d9bd3c34f6b8cbb1d372727071e P 2d88144e1c0548c298e7dd18973eea78: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 01:03:30.124819 23546 heartbeater.cc:503] Master 127.3.75.62:42385 was elected leader, sending a full tablet report...
I20260926 01:03:30.204067 23568 raft_consensus.cc:482] T 21202d9bd3c34f6b8cbb1d372727071e P 2d88144e1c0548c298e7dd18973eea78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:03:30.204198 23568 raft_consensus.cc:504] T 21202d9bd3c34f6b8cbb1d372727071e P 2d88144e1c0548c298e7dd18973eea78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d88144e1c0548c298e7dd18973eea78" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 34807 } } peers { permanent_uuid: "05eb1d0f7a384373b74849d0496fed00" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 32813 } } peers { permanent_uuid: "cc115eb90df347eeb71d0cd9b3e01afb" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 35541 } }
I20260926 01:03:30.204655 23568 leader_election.cc:288] T 21202d9bd3c34f6b8cbb1d372727071e P 2d88144e1c0548c298e7dd18973eea78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05eb1d0f7a384373b74849d0496fed00 (127.3.75.2:32813), cc115eb90df347eeb71d0cd9b3e01afb (127.3.75.1:35541)
I20260926 01:03:30.208694 23556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:30.204863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.3:49109 (local address 127.3.75.2:32813)
0926 01:03:30.204943 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:30.204945 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:30.204955 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:03:30.204992 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:30.204993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:30.205000 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:30.205050 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:03:30.205278 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:30.206030 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:30.207695 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:30.208260 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:30.208333 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:30.208343 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:30.208598 (+ 255us) server_negotiation.cc:299] Negotiation successful
0926 01:03:30.208637 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 01:03:30.208685 23559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:30.204869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.75.3:50671 (local address 127.3.75.1:35541)
0926 01:03:30.204915 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:30.204917 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:30.205064 (+ 147us) server_negotiation.cc:407] Connection header received
0926 01:03:30.205148 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:30.205149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:30.205161 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:30.205205 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:03:30.205368 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:30.206048 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:30.207696 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:30.208258 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:30.208333 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:30.208343 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:30.208555 (+ 212us) server_negotiation.cc:299] Negotiation successful
0926 01:03:30.208601 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 01:03:30.208721 23548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:30.204787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.2:32813 (local address 127.3.75.3:49109)
0926 01:03:30.204835 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:30.204849 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:30.204890 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:30.205097 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:30.205098 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:30.205100 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:03:30.205179 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:30.205181 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:30.206163 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:30.206164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:30.207585 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:30.207589 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:30.208367 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:30.208368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:30.208459 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:30.208465 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 01:03:30.208591 (+ 126us) client_negotiation.cc:240] Negotiation successful
0926 01:03:30.208626 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":73}
I20260926 01:03:30.208736 23569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:30.204791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.1:35541 (local address 127.3.75.3:50671)
0926 01:03:30.204996 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:30.205004 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:30.205074 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:30.205229 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:30.205230 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:30.205231 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:03:30.205293 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:30.205294 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:30.206163 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:30.206164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:30.207585 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:30.207589 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:30.208367 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:30.208368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:30.208459 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:30.208466 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 01:03:30.208541 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 01:03:30.208592 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":93,"thread_start_us":102,"threads_started":1}
I20260926 01:03:30.209312 23368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21202d9bd3c34f6b8cbb1d372727071e" candidate_uuid: "2d88144e1c0548c298e7dd18973eea78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05eb1d0f7a384373b74849d0496fed00" is_pre_election: true
I20260926 01:03:30.209312 23236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21202d9bd3c34f6b8cbb1d372727071e" candidate_uuid: "2d88144e1c0548c298e7dd18973eea78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc115eb90df347eeb71d0cd9b3e01afb" is_pre_election: true
I20260926 01:03:30.209478 23236 raft_consensus.cc:2454] T 21202d9bd3c34f6b8cbb1d372727071e P cc115eb90df347eeb71d0cd9b3e01afb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d88144e1c0548c298e7dd18973eea78 in term 0.
I20260926 01:03:30.209478 23368 raft_consensus.cc:2454] T 21202d9bd3c34f6b8cbb1d372727071e P 05eb1d0f7a384373b74849d0496fed00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d88144e1c0548c298e7dd18973eea78 in term 0.
I20260926 01:03:30.209753 23434 leader_election.cc:302] T 21202d9bd3c34f6b8cbb1d372727071e P 2d88144e1c0548c298e7dd18973eea78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05eb1d0f7a384373b74849d0496fed00, 2d88144e1c0548c298e7dd18973eea78; no voters:
I20260926 01:03:30.209931 23568 raft_consensus.cc:2781] T 21202d9bd3c34f6b8cbb1d372727071e P 2d88144e1c0548c298e7dd18973eea78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:03:30.210037 23568 raft_consensus.cc:482] T 21202d9bd3c34f6b8cbb1d372727071e P 2d88144e1c0548c298e7dd18973eea78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:03:30.210078 23568 raft_consensus.cc:3037] T 21202d9bd3c34f6b8cbb1d372727071e P 2d88144e1c0548c298e7dd18973eea78 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:03:30.210937 23568 raft_consensus.cc:504] T 21202d9bd3c34f6b8cbb1d372727071e P 2d88144e1c0548c298e7dd18973eea78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d88144e1c0548c298e7dd18973eea78" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 34807 } } peers { permanent_uuid: "05eb1d0f7a384373b74849d0496fed00" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 32813 } } peers { permanent_uuid: "cc115eb90df347eeb71d0cd9b3e01afb" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 35541 } }
I20260926 01:03:30.211129 23568 leader_election.cc:288] T 21202d9bd3c34f6b8cbb1d372727071e P 2d88144e1c0548c298e7dd18973eea78 [CANDIDATE]: Term 1 election: Requested vote from peers 05eb1d0f7a384373b74849d0496fed00 (127.3.75.2:32813), cc115eb90df347eeb71d0cd9b3e01afb (127.3.75.1:35541)
I20260926 01:03:30.211301 23236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21202d9bd3c34f6b8cbb1d372727071e" candidate_uuid: "2d88144e1c0548c298e7dd18973eea78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc115eb90df347eeb71d0cd9b3e01afb"
I20260926 01:03:30.211337 23368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21202d9bd3c34f6b8cbb1d372727071e" candidate_uuid: "2d88144e1c0548c298e7dd18973eea78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05eb1d0f7a384373b74849d0496fed00"
I20260926 01:03:30.211385 23236 raft_consensus.cc:3037] T 21202d9bd3c34f6b8cbb1d372727071e P cc115eb90df347eeb71d0cd9b3e01afb [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:03:30.211462 23368 raft_consensus.cc:3037] T 21202d9bd3c34f6b8cbb1d372727071e P 05eb1d0f7a384373b74849d0496fed00 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:03:30.212152 23236 raft_consensus.cc:2454] T 21202d9bd3c34f6b8cbb1d372727071e P cc115eb90df347eeb71d0cd9b3e01afb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d88144e1c0548c298e7dd18973eea78 in term 1.
I20260926 01:03:30.212327 23435 leader_election.cc:302] T 21202d9bd3c34f6b8cbb1d372727071e P 2d88144e1c0548c298e7dd18973eea78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d88144e1c0548c298e7dd18973eea78, cc115eb90df347eeb71d0cd9b3e01afb; no voters:
I20260926 01:03:30.212428 23568 raft_consensus.cc:2781] T 21202d9bd3c34f6b8cbb1d372727071e P 2d88144e1c0548c298e7dd18973eea78 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:03:30.212481 23368 raft_consensus.cc:2454] T 21202d9bd3c34f6b8cbb1d372727071e P 05eb1d0f7a384373b74849d0496fed00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d88144e1c0548c298e7dd18973eea78 in term 1.
I20260926 01:03:30.212574 23568 raft_consensus.cc:686] T 21202d9bd3c34f6b8cbb1d372727071e P 2d88144e1c0548c298e7dd18973eea78 [term 1 LEADER]: Becoming Leader. State: Replica: 2d88144e1c0548c298e7dd18973eea78, State: Running, Role: LEADER
I20260926 01:03:30.212687 23568 consensus_queue.cc:237] T 21202d9bd3c34f6b8cbb1d372727071e P 2d88144e1c0548c298e7dd18973eea78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d88144e1c0548c298e7dd18973eea78" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 34807 } } peers { permanent_uuid: "05eb1d0f7a384373b74849d0496fed00" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 32813 } } peers { permanent_uuid: "cc115eb90df347eeb71d0cd9b3e01afb" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 35541 } }
I20260926 01:03:30.213537 23092 catalog_manager.cc:5495] T 21202d9bd3c34f6b8cbb1d372727071e P 2d88144e1c0548c298e7dd18973eea78 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d88144e1c0548c298e7dd18973eea78 (127.3.75.3). New cstate: current_term: 1 leader_uuid: "2d88144e1c0548c298e7dd18973eea78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d88144e1c0548c298e7dd18973eea78" member_type: VOTER last_known_addr { host: "127.3.75.3" port: 34807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05eb1d0f7a384373b74849d0496fed00" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 32813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc115eb90df347eeb71d0cd9b3e01afb" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 35541 } health_report { overall_health: UNKNOWN } } }
W20260926 01:03:30.216634 23415 tablet.cc:2369] T 21202d9bd3c34f6b8cbb1d372727071e P 05eb1d0f7a384373b74849d0496fed00: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:03:30.230962 23557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:30.228485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34504 (local address 127.3.75.3:34807)
0926 01:03:30.228542 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:30.228544 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:30.228555 (+ 11us) server_negotiation.cc:407] Connection header received
0926 01:03:30.228598 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:30.228599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:30.228618 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:30.228724 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:03:30.228859 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:30.229347 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:30.230137 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:30.230350 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:30.230431 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:30.230587 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:03:30.230692 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:03:30.230791 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:30.230811 (+ 20us) server_negotiation.cc:299] Negotiation successful
0926 01:03:30.230833 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 01:03:30.232717 23236 raft_consensus.cc:1261] T 21202d9bd3c34f6b8cbb1d372727071e P cc115eb90df347eeb71d0cd9b3e01afb [term 1 FOLLOWER]: Refusing update from remote peer 2d88144e1c0548c298e7dd18973eea78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:03:30.232766 23368 raft_consensus.cc:1261] T 21202d9bd3c34f6b8cbb1d372727071e P 05eb1d0f7a384373b74849d0496fed00 [term 1 FOLLOWER]: Refusing update from remote peer 2d88144e1c0548c298e7dd18973eea78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:03:30.233139 23570 consensus_queue.cc:1035] T 21202d9bd3c34f6b8cbb1d372727071e P 2d88144e1c0548c298e7dd18973eea78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc115eb90df347eeb71d0cd9b3e01afb" member_type: VOTER last_known_addr { host: "127.3.75.1" port: 35541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:03:30.233278 23568 consensus_queue.cc:1035] T 21202d9bd3c34f6b8cbb1d372727071e P 2d88144e1c0548c298e7dd18973eea78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05eb1d0f7a384373b74849d0496fed00" member_type: VOTER last_known_addr { host: "127.3.75.2" port: 32813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:03:30.236470 23574 mvcc.cc:205] Tried to move back new op lower bound from 7333415363510546432 to 7333415363431907328. Current Snapshot: MvccSnapshot[applied={T|T < 7333415363510546432}]
I20260926 01:03:30.237646 23091 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:03:30.237739 23091 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:03:30.238519 23091 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ccc552a5d1ec40e69ce6183b57a8a2e8: Sending DeleteTablet for 3 replicas of tablet 21202d9bd3c34f6b8cbb1d372727071e
I20260926 01:03:30.238897 23348 tablet_service.cc:1514] Processing DeleteTablet for tablet 21202d9bd3c34f6b8cbb1d372727071e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:03:30 UTC) from {username='slave'} at 127.0.0.1:49618
I20260926 01:03:30.238941 23480 tablet_service.cc:1514] Processing DeleteTablet for tablet 21202d9bd3c34f6b8cbb1d372727071e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:03:30 UTC) from {username='slave'} at 127.0.0.1:34500
I20260926 01:03:30.239133 23216 tablet_service.cc:1514] Processing DeleteTablet for tablet 21202d9bd3c34f6b8cbb1d372727071e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:03:30 UTC) from {username='slave'} at 127.0.0.1:56804
I20260926 01:03:30.239363 23583 tablet_replica.cc:330] T 21202d9bd3c34f6b8cbb1d372727071e P 05eb1d0f7a384373b74849d0496fed00: stopping tablet replica
I20260926 01:03:30.239367 23585 tablet_replica.cc:330] T 21202d9bd3c34f6b8cbb1d372727071e P cc115eb90df347eeb71d0cd9b3e01afb: stopping tablet replica
I20260926 01:03:30.239449 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 23153
I20260926 01:03:30.239462 23583 raft_consensus.cc:2229] T 21202d9bd3c34f6b8cbb1d372727071e P 05eb1d0f7a384373b74849d0496fed00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:03:30.239539 23585 raft_consensus.cc:2229] T 21202d9bd3c34f6b8cbb1d372727071e P cc115eb90df347eeb71d0cd9b3e01afb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:03:30.239562 23583 raft_consensus.cc:2258] T 21202d9bd3c34f6b8cbb1d372727071e P 05eb1d0f7a384373b74849d0496fed00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:03:30.239665 23585 raft_consensus.cc:2258] T 21202d9bd3c34f6b8cbb1d372727071e P cc115eb90df347eeb71d0cd9b3e01afb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:03:30.239705 23584 tablet_replica.cc:330] T 21202d9bd3c34f6b8cbb1d372727071e P 2d88144e1c0548c298e7dd18973eea78: stopping tablet replica
I20260926 01:03:30.239949 23584 raft_consensus.cc:2229] T 21202d9bd3c34f6b8cbb1d372727071e P 2d88144e1c0548c298e7dd18973eea78 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:03:30.240047 23584 raft_consensus.cc:2258] T 21202d9bd3c34f6b8cbb1d372727071e P 2d88144e1c0548c298e7dd18973eea78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:03:30.240542 23583 ts_tablet_manager.cc:1905] T 21202d9bd3c34f6b8cbb1d372727071e P 05eb1d0f7a384373b74849d0496fed00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:03:30.240635 23584 ts_tablet_manager.cc:1905] T 21202d9bd3c34f6b8cbb1d372727071e P 2d88144e1c0548c298e7dd18973eea78: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:03:30.242443 23583 ts_tablet_manager.cc:1918] T 21202d9bd3c34f6b8cbb1d372727071e P 05eb1d0f7a384373b74849d0496fed00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:03:30.242448 23584 ts_tablet_manager.cc:1918] T 21202d9bd3c34f6b8cbb1d372727071e P 2d88144e1c0548c298e7dd18973eea78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:03:30.242518 23583 log.cc:1196] T 21202d9bd3c34f6b8cbb1d372727071e P 05eb1d0f7a384373b74849d0496fed00: Deleting WAL directory at /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/ts-1/wal/wals/21202d9bd3c34f6b8cbb1d372727071e
I20260926 01:03:30.242584 23584 log.cc:1196] T 21202d9bd3c34f6b8cbb1d372727071e P 2d88144e1c0548c298e7dd18973eea78: Deleting WAL directory at /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384372073861-3372-0/minicluster-data/ts-2/wal/wals/21202d9bd3c34f6b8cbb1d372727071e
I20260926 01:03:30.242797 23583 ts_tablet_manager.cc:1939] T 21202d9bd3c34f6b8cbb1d372727071e P 05eb1d0f7a384373b74849d0496fed00: Deleting consensus metadata
I20260926 01:03:30.242930 23584 ts_tablet_manager.cc:1939] T 21202d9bd3c34f6b8cbb1d372727071e P 2d88144e1c0548c298e7dd18973eea78: Deleting consensus metadata
I20260926 01:03:30.243211 23078 catalog_manager.cc:4841] TS 05eb1d0f7a384373b74849d0496fed00 (127.3.75.2:32813): tablet 21202d9bd3c34f6b8cbb1d372727071e (table test-table [id=6728aa05763f458fa054d4b53109f90b]) successfully deleted
I20260926 01:03:30.243517 23079 catalog_manager.cc:4841] TS 2d88144e1c0548c298e7dd18973eea78 (127.3.75.3:34807): tablet 21202d9bd3c34f6b8cbb1d372727071e (table test-table [id=6728aa05763f458fa054d4b53109f90b]) successfully deleted
W20260926 01:03:30.246981 23079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.75.1:35541 (error 108)
I20260926 01:03:30.247543 23558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:30.247361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.1:35541 (local address 127.0.0.1:56818)
0926 01:03:30.247425 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:30.247473 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.1:35541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 01:03:30.247541 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 23285
W20260926 01:03:30.249441 23079 catalog_manager.cc:4584] TS cc115eb90df347eeb71d0cd9b3e01afb (127.3.75.1:35541): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 21202d9bd3c34f6b8cbb1d372727071e: Network error: Client connection negotiation failed: client connection to 127.3.75.1:35541: connect: Connection refused (error 111)
I20260926 01:03:30.252601 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 23417
I20260926 01:03:30.257361 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 23061
2026-09-26T01:03:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (736 ms)
[----------] 6 tests from EncryptionPolicyTest (7882 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384372073861-3372-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:03:30 dist-test-slave-7v10 krb5kdc[23589](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:03:30 dist-test-slave-7v10 krb5kdc[23589](info): set up 2 sockets
Sep 26 01:03:30 dist-test-slave-7v10 krb5kdc[23589](info): commencing operation
krb5kdc: starting...
W20260926 01:03:31.293812 3372 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Sep 26 01:03:31 dist-test-slave-7v10 krb5kdc[23589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:03:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:03:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384372073861-3372-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384372073861-3372-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 01:03:31.413350 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384372073861-3372-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384372073861-3372-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384372073861-3372-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42013
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384372073861-3372-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42013
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:03:31.493183 23603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:03:31.493431 23603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:03:31.493471 23603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:03:31.494724 23603 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:03:31.494796 23603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:03:31.494820 23603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:03:31.494853 23603 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:03:31.494911 23603 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:03:31.496397 23603 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384372073861-3372-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384372073861-3372-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42013
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384372073861-3372-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384372073861-3372-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:03:31.496678 23603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:03:31.496919 23603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:03:31.501951 23611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:31.501964 23608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:31.501971 23609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:03:31.502146 23603 server_base.cc:1034] running on GCE node
I20260926 01:03:31.502497 23603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:03:31.502839 23603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:03:31.504037 23603 hybrid_clock.cc:648] HybridClock initialized: now 1790384611504016 us; error 37 us; skew 500 ppm
Sep 26 01:03:31 dist-test-slave-7v10 krb5kdc[23589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384611, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:03:31.506425 23603 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 01:03:31.506969 23603 webserver.cc:460] Webserver started at http://127.0.0.1:34169/ using document root <none> and password file <none>
I20260926 01:03:31.507229 23603 fs_manager.cc:359] Metadata directory not provided
I20260926 01:03:31.507304 23603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:03:31.507463 23603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:03:31.508514 23603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384372073861-3372-0/minicluster-data/master-0/data/instance:
uuid: "a7610b40f17041ad8158484b38784bc0"
format_stamp: "Formatted at 2026-09-26 01:03:31 on dist-test-slave-7v10"
server_key: "9fd6e143224f5f9d1b56ad9a83f3302c"
server_key_iv: "b9b7b96b851f719d8147b1da3db41f99"
server_key_version: "encryptionkey@0"
I20260926 01:03:31.508880 23603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384372073861-3372-0/minicluster-data/master-0/wal/instance:
uuid: "a7610b40f17041ad8158484b38784bc0"
format_stamp: "Formatted at 2026-09-26 01:03:31 on dist-test-slave-7v10"
server_key: "9fd6e143224f5f9d1b56ad9a83f3302c"
server_key_iv: "b9b7b96b851f719d8147b1da3db41f99"
server_key_version: "encryptionkey@0"
I20260926 01:03:31.510174 23603 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 01:03:31.510913 23618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:31.511073 23603 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 01:03:31.511183 23603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384372073861-3372-0/minicluster-data/master-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384372073861-3372-0/minicluster-data/master-0/wal
uuid: "a7610b40f17041ad8158484b38784bc0"
format_stamp: "Formatted at 2026-09-26 01:03:31 on dist-test-slave-7v10"
server_key: "9fd6e143224f5f9d1b56ad9a83f3302c"
server_key_iv: "b9b7b96b851f719d8147b1da3db41f99"
server_key_version: "encryptionkey@0"
I20260926 01:03:31.511274 23603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384372073861-3372-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384372073861-3372-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384372073861-3372-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:03:31.533434 23603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:03:31.535342 23603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:03:31.535522 23603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:03:31.539680 23603 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42013
I20260926 01:03:31.539717 23670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42013 every 8 connection(s)
I20260926 01:03:31.540064 23603 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
I20260926 01:03:31.541119 23671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:03:31.543457 23671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7610b40f17041ad8158484b38784bc0: Bootstrap starting.
I20260926 01:03:31.544060 23671 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7610b40f17041ad8158484b38784bc0: Neither blocks nor log segments found. Creating new log.
I20260926 01:03:31.544394 23671 log.cc:824] T 00000000000000000000000000000000 P a7610b40f17041ad8158484b38784bc0: Log is configured to *not* fsync() on all Append() calls
I20260926 01:03:31.545081 23671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7610b40f17041ad8158484b38784bc0: No bootstrap required, opened a new log
I20260926 01:03:31.546674 23671 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7610b40f17041ad8158484b38784bc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7610b40f17041ad8158484b38784bc0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42013 } }
I20260926 01:03:31.546813 23671 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7610b40f17041ad8158484b38784bc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:03:31.546835 23671 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7610b40f17041ad8158484b38784bc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7610b40f17041ad8158484b38784bc0, State: Initialized, Role: FOLLOWER
I20260926 01:03:31.547050 23671 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7610b40f17041ad8158484b38784bc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7610b40f17041ad8158484b38784bc0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42013 } }
I20260926 01:03:31.547119 23671 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7610b40f17041ad8158484b38784bc0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:03:31.547163 23671 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7610b40f17041ad8158484b38784bc0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:03:31.547271 23671 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7610b40f17041ad8158484b38784bc0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:03:31.547801 23671 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7610b40f17041ad8158484b38784bc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7610b40f17041ad8158484b38784bc0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42013 } }
I20260926 01:03:31.547917 23671 leader_election.cc:302] T 00000000000000000000000000000000 P a7610b40f17041ad8158484b38784bc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7610b40f17041ad8158484b38784bc0; no voters:
I20260926 01:03:31.548105 23671 leader_election.cc:288] T 00000000000000000000000000000000 P a7610b40f17041ad8158484b38784bc0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:03:31.548127 23674 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7610b40f17041ad8158484b38784bc0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:03:31.548251 23674 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7610b40f17041ad8158484b38784bc0 [term 1 LEADER]: Becoming Leader. State: Replica: a7610b40f17041ad8158484b38784bc0, State: Running, Role: LEADER
I20260926 01:03:31.548269 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 23603
I20260926 01:03:31.548382 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384372073861-3372-0/minicluster-data/master-0/wal/instance
I20260926 01:03:31.548382 23671 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7610b40f17041ad8158484b38784bc0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:03:31.548399 23674 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7610b40f17041ad8158484b38784bc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7610b40f17041ad8158484b38784bc0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42013 } }
I20260926 01:03:31.548557 3372 external_mini_cluster.cc:1428] Setting key b5fccb69086575b7317c87b0a9d91a06
I20260926 01:03:31.548725 23675 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7610b40f17041ad8158484b38784bc0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7610b40f17041ad8158484b38784bc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7610b40f17041ad8158484b38784bc0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42013 } } }
I20260926 01:03:31.548810 23675 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7610b40f17041ad8158484b38784bc0 [sys.catalog]: This master's current role is: LEADER
I20260926 01:03:31.548861 23676 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7610b40f17041ad8158484b38784bc0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7610b40f17041ad8158484b38784bc0. Latest consensus state: current_term: 1 leader_uuid: "a7610b40f17041ad8158484b38784bc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7610b40f17041ad8158484b38784bc0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42013 } } }
I20260926 01:03:31.548946 23676 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7610b40f17041ad8158484b38784bc0 [sys.catalog]: This master's current role is: LEADER
W20260926 01:03:31.550153 23691 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a7610b40f17041ad8158484b38784bc0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:03:31.550241 23691 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:03:31.550341 23680 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:03:31.550540 23680 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:03:31.552014 23680 catalog_manager.cc:1347] Generated new cluster ID: 8e46592fac254d3eb633f38590b72a61
I20260926 01:03:31.552065 23680 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 01:03:31 dist-test-slave-7v10 krb5kdc[23589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 01:03:31.556281 23680 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:03:31.557248 23680 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:03:31.560614 23685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:31.549243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46399 (local address 127.0.0.1:42013)
0926 01:03:31.549455 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:31.549464 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:31.549496 (+ 32us) server_negotiation.cc:407] Connection header received
0926 01:03:31.549608 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:31.549615 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:31.549694 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:31.550707 (+ 1013us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:31.550797 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:31.552224 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:31.553312 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:31.553934 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:31.554064 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:31.556885 (+ 2821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:31.556889 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:31.556889 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:03:31.556900 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:31.558795 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:31.559286 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:31.559289 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:31.559289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:31.559390 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:31.559644 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:31.559645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:31.559645 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:31.559897 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:31.560050 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:31.560200 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 01:03:31.560404 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":107,"threads_started":1}
I20260926 01:03:31.561599 3372 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:03:31.562256 23680 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7610b40f17041ad8158484b38784bc0: Generated new TSK 0
I20260926 01:03:31.562404 23680 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:03:31.569572 23685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:31.563150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59937 (local address 127.0.0.1:42013)
0926 01:03:31.563186 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:31.563187 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:31.563243 (+ 56us) server_negotiation.cc:407] Connection header received
0926 01:03:31.563310 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:31.563311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:31.563319 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:31.563359 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:31.563974 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:31.564459 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:31.565508 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:31.565698 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:31.565748 (+ 50us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:31.567078 (+ 1330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:31.567079 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:31.567079 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:03:31.567085 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:31.568471 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:31.568879 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:31.568880 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:31.568880 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:31.568926 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:31.569143 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:31.569143 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:31.569143 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:31.569241 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:31.569311 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:31.569470 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 01:03:31.569515 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 01:03:31.571955 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 23603
W20260926 01:03:31.575941 23587 connection.cc:535] client connection to 127.0.0.1:42013 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42013): Connection reset by peer (error 104)
2026-09-26T01:03:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1312 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384372073861-3372-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:03:31 dist-test-slave-7v10 krb5kdc[23702](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:03:31 dist-test-slave-7v10 krb5kdc[23702](info): set up 2 sockets
Sep 26 01:03:31 dist-test-slave-7v10 krb5kdc[23702](info): commencing operation
krb5kdc: starting...
W20260926 01:03:32.666376 3372 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Sep 26 01:03:32 dist-test-slave-7v10 krb5kdc[23702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:03:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:03:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384372073861-3372-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384372073861-3372-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 01:03:32.786988 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384372073861-3372-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384372073861-3372-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384372073861-3372-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45815
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384372073861-3372-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:03:32.868397 23715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:03:32.868597 23715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:03:32.868621 23715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:03:32.869799 23715 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:03:32.869841 23715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:03:32.869856 23715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:03:32.869869 23715 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:03:32.869881 23715 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:03:32.871403 23715 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384372073861-3372-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384372073861-3372-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45815
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384372073861-3372-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384372073861-3372-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:03:32.871649 23715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:03:32.871891 23715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:03:32.877316 23721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:32.877293 23720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:03:32.877410 23715 server_base.cc:1034] running on GCE node
W20260926 01:03:32.877311 23723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:03:32.877830 23715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:03:32.878140 23715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:03:32.879315 23715 hybrid_clock.cc:648] HybridClock initialized: now 1790384612879292 us; error 39 us; skew 500 ppm
Sep 26 01:03:32 dist-test-slave-7v10 krb5kdc[23702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384612, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:03:32.886730 23715 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 01:03:32.887271 23715 webserver.cc:460] Webserver started at http://127.0.0.1:32781/ using document root <none> and password file <none>
I20260926 01:03:32.887523 23715 fs_manager.cc:359] Metadata directory not provided
I20260926 01:03:32.887596 23715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:03:32.887777 23715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:03:32.888831 23715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384372073861-3372-0/minicluster-data/master-0/data/instance:
uuid: "aa43c5a901724a4e8f253ab4e65430a4"
format_stamp: "Formatted at 2026-09-26 01:03:32 on dist-test-slave-7v10"
server_key: "bcfacababf7d966c04d9409ae9001fd1"
server_key_iv: "9a5a1f7c781982624c8908c481565cfe"
server_key_version: "encryptionkey@0"
I20260926 01:03:32.889209 23715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384372073861-3372-0/minicluster-data/master-0/wal/instance:
uuid: "aa43c5a901724a4e8f253ab4e65430a4"
format_stamp: "Formatted at 2026-09-26 01:03:32 on dist-test-slave-7v10"
server_key: "bcfacababf7d966c04d9409ae9001fd1"
server_key_iv: "9a5a1f7c781982624c8908c481565cfe"
server_key_version: "encryptionkey@0"
I20260926 01:03:32.890532 23715 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 01:03:32.891350 23730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:32.891523 23715 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:32.891629 23715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384372073861-3372-0/minicluster-data/master-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384372073861-3372-0/minicluster-data/master-0/wal
uuid: "aa43c5a901724a4e8f253ab4e65430a4"
format_stamp: "Formatted at 2026-09-26 01:03:32 on dist-test-slave-7v10"
server_key: "bcfacababf7d966c04d9409ae9001fd1"
server_key_iv: "9a5a1f7c781982624c8908c481565cfe"
server_key_version: "encryptionkey@0"
I20260926 01:03:32.891727 23715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384372073861-3372-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384372073861-3372-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384372073861-3372-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:03:32.901012 23715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:03:32.902732 23715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:03:32.902962 23715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:03:32.906840 23715 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45815
I20260926 01:03:32.906911 23783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45815 every 8 connection(s)
I20260926 01:03:32.907275 23715 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
I20260926 01:03:32.908218 23784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:03:32.910356 23784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa43c5a901724a4e8f253ab4e65430a4: Bootstrap starting.
I20260926 01:03:32.910975 23784 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa43c5a901724a4e8f253ab4e65430a4: Neither blocks nor log segments found. Creating new log.
I20260926 01:03:32.911280 23784 log.cc:824] T 00000000000000000000000000000000 P aa43c5a901724a4e8f253ab4e65430a4: Log is configured to *not* fsync() on all Append() calls
I20260926 01:03:32.911715 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 23715
I20260926 01:03:32.911837 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384372073861-3372-0/minicluster-data/master-0/wal/instance
I20260926 01:03:32.912047 3372 external_mini_cluster.cc:1428] Setting key 96d0e0909557bc462ef36ab0c32a35fb
I20260926 01:03:32.912127 23784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa43c5a901724a4e8f253ab4e65430a4: No bootstrap required, opened a new log
I20260926 01:03:32.914633 23784 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa43c5a901724a4e8f253ab4e65430a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa43c5a901724a4e8f253ab4e65430a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45815 } }
I20260926 01:03:32.914837 23784 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa43c5a901724a4e8f253ab4e65430a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:03:32.914902 23784 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa43c5a901724a4e8f253ab4e65430a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa43c5a901724a4e8f253ab4e65430a4, State: Initialized, Role: FOLLOWER
I20260926 01:03:32.915045 23784 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa43c5a901724a4e8f253ab4e65430a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa43c5a901724a4e8f253ab4e65430a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45815 } }
I20260926 01:03:32.915128 23784 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa43c5a901724a4e8f253ab4e65430a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:03:32.915158 23784 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa43c5a901724a4e8f253ab4e65430a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:03:32.915220 23784 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa43c5a901724a4e8f253ab4e65430a4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:03:32.915890 23784 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa43c5a901724a4e8f253ab4e65430a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa43c5a901724a4e8f253ab4e65430a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45815 } }
I20260926 01:03:32.915992 23784 leader_election.cc:302] T 00000000000000000000000000000000 P aa43c5a901724a4e8f253ab4e65430a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa43c5a901724a4e8f253ab4e65430a4; no voters:
I20260926 01:03:32.916177 23784 leader_election.cc:288] T 00000000000000000000000000000000 P aa43c5a901724a4e8f253ab4e65430a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:03:32.916198 23789 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa43c5a901724a4e8f253ab4e65430a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:03:32.916313 23789 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa43c5a901724a4e8f253ab4e65430a4 [term 1 LEADER]: Becoming Leader. State: Replica: aa43c5a901724a4e8f253ab4e65430a4, State: Running, Role: LEADER
I20260926 01:03:32.916433 23784 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa43c5a901724a4e8f253ab4e65430a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:03:32.916450 23789 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa43c5a901724a4e8f253ab4e65430a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa43c5a901724a4e8f253ab4e65430a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45815 } }
I20260926 01:03:32.916836 23790 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa43c5a901724a4e8f253ab4e65430a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa43c5a901724a4e8f253ab4e65430a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa43c5a901724a4e8f253ab4e65430a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45815 } } }
I20260926 01:03:32.916944 23790 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa43c5a901724a4e8f253ab4e65430a4 [sys.catalog]: This master's current role is: LEADER
I20260926 01:03:32.916817 23789 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa43c5a901724a4e8f253ab4e65430a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa43c5a901724a4e8f253ab4e65430a4. Latest consensus state: current_term: 1 leader_uuid: "aa43c5a901724a4e8f253ab4e65430a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa43c5a901724a4e8f253ab4e65430a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45815 } } }
I20260926 01:03:32.917263 23789 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa43c5a901724a4e8f253ab4e65430a4 [sys.catalog]: This master's current role is: LEADER
I20260926 01:03:32.917599 23795 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:03:32.918171 23795 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:03:32.918359 23788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:32.912556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46085 (local address 127.0.0.1:45815)
0926 01:03:32.912736 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:32.912743 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:32.912769 (+ 26us) server_negotiation.cc:407] Connection header received
0926 01:03:32.913402 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:32.913408 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:32.913452 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:32.913559 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:32.913829 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:32.914698 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:32.915891 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:32.916531 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:32.916962 (+ 431us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:32.917438 (+ 476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:32.917442 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:32.917442 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:03:32.917644 (+ 202us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:32.917698 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:03:32.917714 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:32.918033 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:32.918079 (+ 46us) server_negotiation.cc:299] Negotiation successful
0926 01:03:32.918108 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":86,"threads_started":1}
I20260926 01:03:32.919423 3372 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:03:32.919835 23795 catalog_manager.cc:1347] Generated new cluster ID: e8cdd03ba5e64f968786b78277477839
I20260926 01:03:32.919888 23795 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:03:32.930094 23788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:32.920864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57191 (local address 127.0.0.1:45815)
0926 01:03:32.920900 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:32.920901 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:32.927241 (+ 6340us) server_negotiation.cc:407] Connection header received
0926 01:03:32.927305 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:32.927306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:32.927315 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:32.927369 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:32.927513 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:32.927999 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:32.928739 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:32.928965 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:32.929060 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:32.929797 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:32.929798 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:32.929798 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:03:32.929802 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:32.929816 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:03:32.929820 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:32.929874 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:32.929985 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 01:03:32.929995 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":17}
I20260926 01:03:32.942294 23795 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:03:32.942794 23795 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:03:32.947407 23795 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa43c5a901724a4e8f253ab4e65430a4: Generated new TSK 0
I20260926 01:03:32.947597 23795 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:03:32.951320 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 23715
W20260926 01:03:32.956099 23700 connection.cc:535] client connection to 127.0.0.1:45815 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45815): Connection reset by peer (error 104)
2026-09-26T01:03:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1380 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384372073861-3372-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:03:33 dist-test-slave-7v10 krb5kdc[23816](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:03:33 dist-test-slave-7v10 krb5kdc[23816](info): set up 2 sockets
Sep 26 01:03:33 dist-test-slave-7v10 krb5kdc[23816](info): commencing operation
krb5kdc: starting...
W20260926 01:03:34.003394 3372 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Sep 26 01:03:34 dist-test-slave-7v10 krb5kdc[23816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:03:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:03:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384372073861-3372-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384372073861-3372-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 01:03:34.122085 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384372073861-3372-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384372073861-3372-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384372073861-3372-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46797
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384372073861-3372-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46797
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:03:34.203583 23829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:03:34.203838 23829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:03:34.203878 23829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:03:34.205289 23829 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:03:34.205395 23829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:03:34.205427 23829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:03:34.205458 23829 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:03:34.205483 23829 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:03:34.207319 23829 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384372073861-3372-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384372073861-3372-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46797
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384372073861-3372-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384372073861-3372-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:03:34.207617 23829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:03:34.207862 23829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:03:34.212865 23835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:03:34.212975 23829 server_base.cc:1034] running on GCE node
W20260926 01:03:34.212850 23834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:34.212852 23837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:03:34.213343 23829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:03:34.213598 23829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:03:34.214740 23829 hybrid_clock.cc:648] HybridClock initialized: now 1790384614214718 us; error 38 us; skew 500 ppm
Sep 26 01:03:34 dist-test-slave-7v10 krb5kdc[23816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384614, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:03:34.217136 23829 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 01:03:34.217561 23829 webserver.cc:460] Webserver started at http://127.0.0.1:39263/ using document root <none> and password file <none>
I20260926 01:03:34.217799 23829 fs_manager.cc:359] Metadata directory not provided
I20260926 01:03:34.217875 23829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:03:34.218020 23829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:03:34.219406 23829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384372073861-3372-0/minicluster-data/master-0/data/instance:
uuid: "282f575cf073467d9a5bde86cc1e5c1d"
format_stamp: "Formatted at 2026-09-26 01:03:34 on dist-test-slave-7v10"
server_key: "59de7c9a10a67bc7d9068fe2e0c5c993"
server_key_iv: "6ca4b02a297258c777872147cc23120a"
server_key_version: "encryptionkey@0"
I20260926 01:03:34.219825 23829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384372073861-3372-0/minicluster-data/master-0/wal/instance:
uuid: "282f575cf073467d9a5bde86cc1e5c1d"
format_stamp: "Formatted at 2026-09-26 01:03:34 on dist-test-slave-7v10"
server_key: "59de7c9a10a67bc7d9068fe2e0c5c993"
server_key_iv: "6ca4b02a297258c777872147cc23120a"
server_key_version: "encryptionkey@0"
I20260926 01:03:34.221203 23829 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:03:34.221964 23844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:34.222143 23829 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:03:34.222252 23829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384372073861-3372-0/minicluster-data/master-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384372073861-3372-0/minicluster-data/master-0/wal
uuid: "282f575cf073467d9a5bde86cc1e5c1d"
format_stamp: "Formatted at 2026-09-26 01:03:34 on dist-test-slave-7v10"
server_key: "59de7c9a10a67bc7d9068fe2e0c5c993"
server_key_iv: "6ca4b02a297258c777872147cc23120a"
server_key_version: "encryptionkey@0"
I20260926 01:03:34.222335 23829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384372073861-3372-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384372073861-3372-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384372073861-3372-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:03:34.225713 23829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:03:34.227670 23829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:03:34.227838 23829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:03:34.231714 23829 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46797
I20260926 01:03:34.231743 23897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46797 every 8 connection(s)
I20260926 01:03:34.232074 23829 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
I20260926 01:03:34.233032 23898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:03:34.235384 23898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 282f575cf073467d9a5bde86cc1e5c1d: Bootstrap starting.
I20260926 01:03:34.236030 23898 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 282f575cf073467d9a5bde86cc1e5c1d: Neither blocks nor log segments found. Creating new log.
I20260926 01:03:34.236326 23898 log.cc:824] T 00000000000000000000000000000000 P 282f575cf073467d9a5bde86cc1e5c1d: Log is configured to *not* fsync() on all Append() calls
I20260926 01:03:34.236955 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 23829
I20260926 01:03:34.237051 23898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 282f575cf073467d9a5bde86cc1e5c1d: No bootstrap required, opened a new log
I20260926 01:03:34.237082 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384372073861-3372-0/minicluster-data/master-0/wal/instance
I20260926 01:03:34.237267 3372 external_mini_cluster.cc:1428] Setting key 73f456b03a8c51edf32ca5c8caefe3b9
I20260926 01:03:34.239606 23898 raft_consensus.cc:348] T 00000000000000000000000000000000 P 282f575cf073467d9a5bde86cc1e5c1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "282f575cf073467d9a5bde86cc1e5c1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46797 } }
I20260926 01:03:34.239785 23898 raft_consensus.cc:374] T 00000000000000000000000000000000 P 282f575cf073467d9a5bde86cc1e5c1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:03:34.239832 23898 raft_consensus.cc:729] T 00000000000000000000000000000000 P 282f575cf073467d9a5bde86cc1e5c1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 282f575cf073467d9a5bde86cc1e5c1d, State: Initialized, Role: FOLLOWER
I20260926 01:03:34.239979 23902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:34.237808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54181 (local address 127.0.0.1:46797)
0926 01:03:34.237975 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:34.237983 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:34.238009 (+ 26us) server_negotiation.cc:407] Connection header received
0926 01:03:34.238491 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:34.238497 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:34.238523 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:34.238643 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:34.239335 (+ 692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:34.239338 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:34.239338 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:03:34.239342 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:34.239377 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:03:34.239387 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:34.239443 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:34.239528 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 01:03:34.239588 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":84,"threads_started":1}
I20260926 01:03:34.239955 23898 consensus_queue.cc:260] T 00000000000000000000000000000000 P 282f575cf073467d9a5bde86cc1e5c1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "282f575cf073467d9a5bde86cc1e5c1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46797 } }
I20260926 01:03:34.240056 23898 raft_consensus.cc:388] T 00000000000000000000000000000000 P 282f575cf073467d9a5bde86cc1e5c1d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:03:34.240096 23898 raft_consensus.cc:482] T 00000000000000000000000000000000 P 282f575cf073467d9a5bde86cc1e5c1d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:03:34.240154 23898 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 282f575cf073467d9a5bde86cc1e5c1d [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:03:34.240864 3372 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:03:34.240898 23898 raft_consensus.cc:504] T 00000000000000000000000000000000 P 282f575cf073467d9a5bde86cc1e5c1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "282f575cf073467d9a5bde86cc1e5c1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46797 } }
I20260926 01:03:34.241029 23898 leader_election.cc:302] T 00000000000000000000000000000000 P 282f575cf073467d9a5bde86cc1e5c1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 282f575cf073467d9a5bde86cc1e5c1d; no voters:
I20260926 01:03:34.241218 23898 leader_election.cc:288] T 00000000000000000000000000000000 P 282f575cf073467d9a5bde86cc1e5c1d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:03:34.241242 23903 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 282f575cf073467d9a5bde86cc1e5c1d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:03:34.241359 23903 raft_consensus.cc:686] T 00000000000000000000000000000000 P 282f575cf073467d9a5bde86cc1e5c1d [term 1 LEADER]: Becoming Leader. State: Replica: 282f575cf073467d9a5bde86cc1e5c1d, State: Running, Role: LEADER
I20260926 01:03:34.241480 23898 sys_catalog.cc:564] T 00000000000000000000000000000000 P 282f575cf073467d9a5bde86cc1e5c1d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:03:34.241488 23903 consensus_queue.cc:237] T 00000000000000000000000000000000 P 282f575cf073467d9a5bde86cc1e5c1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "282f575cf073467d9a5bde86cc1e5c1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46797 } }
I20260926 01:03:34.241922 23905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 282f575cf073467d9a5bde86cc1e5c1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 282f575cf073467d9a5bde86cc1e5c1d. Latest consensus state: current_term: 1 leader_uuid: "282f575cf073467d9a5bde86cc1e5c1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "282f575cf073467d9a5bde86cc1e5c1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46797 } } }
I20260926 01:03:34.242007 23905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 282f575cf073467d9a5bde86cc1e5c1d [sys.catalog]: This master's current role is: LEADER
I20260926 01:03:34.242244 23904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 282f575cf073467d9a5bde86cc1e5c1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "282f575cf073467d9a5bde86cc1e5c1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "282f575cf073467d9a5bde86cc1e5c1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46797 } } }
I20260926 01:03:34.242549 23904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 282f575cf073467d9a5bde86cc1e5c1d [sys.catalog]: This master's current role is: LEADER
I20260926 01:03:34.242713 23913 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:03:34.243273 23913 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:03:34.244390 23902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:34.243158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37117 (local address 127.0.0.1:46797)
0926 01:03:34.243194 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:34.243196 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:34.243331 (+ 135us) server_negotiation.cc:407] Connection header received
0926 01:03:34.243420 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:34.243422 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:34.243427 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:34.243457 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:34.244145 (+ 688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:34.244146 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:34.244146 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:03:34.244158 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:34.244170 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:03:34.244174 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:34.244244 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:34.244302 (+ 58us) server_negotiation.cc:299] Negotiation successful
0926 01:03:34.244317 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 01:03:34.244966 23913 catalog_manager.cc:1347] Generated new cluster ID: c61a7890762a493aa331e264cfc63d4a
I20260926 01:03:34.245051 23913 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:03:34.258016 23913 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:03:34.258244 23913 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:03:34.261359 23913 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 282f575cf073467d9a5bde86cc1e5c1d: Generated new TSK 0
I20260926 01:03:34.261538 23913 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:03:34.267583 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 23829
2026-09-26T01:03:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1316 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384372073861-3372-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:03:34 dist-test-slave-7v10 krb5kdc[23929](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:03:34 dist-test-slave-7v10 krb5kdc[23929](info): set up 2 sockets
Sep 26 01:03:34 dist-test-slave-7v10 krb5kdc[23929](info): commencing operation
krb5kdc: starting...
W20260926 01:03:35.298355 3372 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Sep 26 01:03:35 dist-test-slave-7v10 krb5kdc[23929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:03:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:03:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384372073861-3372-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384372073861-3372-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 01:03:35.417436 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384372073861-3372-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384372073861-3372-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384372073861-3372-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44921
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384372073861-3372-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:03:35.496124 23942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:03:35.496309 23942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:03:35.496330 23942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:03:35.497390 23942 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:03:35.497427 23942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:03:35.497440 23942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:03:35.497490 23942 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:03:35.497512 23942 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:03:35.498939 23942 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384372073861-3372-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384372073861-3372-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384372073861-3372-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384372073861-3372-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:03:35.499166 23942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:03:35.499388 23942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:03:35.504041 23949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:35.504042 23948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:35.504122 23951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:03:35.504163 23942 server_base.cc:1034] running on GCE node
I20260926 01:03:35.504422 23942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:03:35.504716 23942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:03:35.505887 23942 hybrid_clock.cc:648] HybridClock initialized: now 1790384615505874 us; error 30 us; skew 500 ppm
Sep 26 01:03:35 dist-test-slave-7v10 krb5kdc[23929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384615, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:03:35.508299 23942 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 01:03:35.508693 23942 webserver.cc:460] Webserver started at http://127.0.0.1:37023/ using document root <none> and password file <none>
I20260926 01:03:35.508904 23942 fs_manager.cc:359] Metadata directory not provided
I20260926 01:03:35.508966 23942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:03:35.509102 23942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:03:35.510098 23942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384372073861-3372-0/minicluster-data/master-0/data/instance:
uuid: "f6caf52d00a14c5bbeb443c5ee16a521"
format_stamp: "Formatted at 2026-09-26 01:03:35 on dist-test-slave-7v10"
server_key: "4d6fe5eb02f393bbac76d9f9f364c293"
server_key_iv: "6cb87d50a9d46ec327c9acd68c4cbb00"
server_key_version: "encryptionkey@0"
I20260926 01:03:35.510452 23942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384372073861-3372-0/minicluster-data/master-0/wal/instance:
uuid: "f6caf52d00a14c5bbeb443c5ee16a521"
format_stamp: "Formatted at 2026-09-26 01:03:35 on dist-test-slave-7v10"
server_key: "4d6fe5eb02f393bbac76d9f9f364c293"
server_key_iv: "6cb87d50a9d46ec327c9acd68c4cbb00"
server_key_version: "encryptionkey@0"
I20260926 01:03:35.511679 23942 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 01:03:35.512324 23958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:35.512465 23942 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 01:03:35.512549 23942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384372073861-3372-0/minicluster-data/master-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384372073861-3372-0/minicluster-data/master-0/wal
uuid: "f6caf52d00a14c5bbeb443c5ee16a521"
format_stamp: "Formatted at 2026-09-26 01:03:35 on dist-test-slave-7v10"
server_key: "4d6fe5eb02f393bbac76d9f9f364c293"
server_key_iv: "6cb87d50a9d46ec327c9acd68c4cbb00"
server_key_version: "encryptionkey@0"
I20260926 01:03:35.512617 23942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384372073861-3372-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384372073861-3372-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384372073861-3372-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:03:35.527709 23942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:03:35.530014 23942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:03:35.530172 23942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:03:35.534207 23942 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44921
I20260926 01:03:35.534251 24010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44921 every 8 connection(s)
I20260926 01:03:35.534605 23942 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
I20260926 01:03:35.535591 24011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:03:35.537789 24011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6caf52d00a14c5bbeb443c5ee16a521: Bootstrap starting.
I20260926 01:03:35.538386 24011 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6caf52d00a14c5bbeb443c5ee16a521: Neither blocks nor log segments found. Creating new log.
I20260926 01:03:35.538662 24011 log.cc:824] T 00000000000000000000000000000000 P f6caf52d00a14c5bbeb443c5ee16a521: Log is configured to *not* fsync() on all Append() calls
I20260926 01:03:35.539410 24011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6caf52d00a14c5bbeb443c5ee16a521: No bootstrap required, opened a new log
I20260926 01:03:35.541057 24011 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6caf52d00a14c5bbeb443c5ee16a521 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6caf52d00a14c5bbeb443c5ee16a521" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44921 } }
I20260926 01:03:35.541195 24011 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6caf52d00a14c5bbeb443c5ee16a521 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:03:35.541249 24011 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6caf52d00a14c5bbeb443c5ee16a521 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6caf52d00a14c5bbeb443c5ee16a521, State: Initialized, Role: FOLLOWER
I20260926 01:03:35.541407 24011 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6caf52d00a14c5bbeb443c5ee16a521 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6caf52d00a14c5bbeb443c5ee16a521" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44921 } }
I20260926 01:03:35.541471 24011 raft_consensus.cc:388] T 00000000000000000000000000000000 P f6caf52d00a14c5bbeb443c5ee16a521 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:03:35.541507 24011 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6caf52d00a14c5bbeb443c5ee16a521 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:03:35.541559 24011 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6caf52d00a14c5bbeb443c5ee16a521 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:03:35.542188 24011 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6caf52d00a14c5bbeb443c5ee16a521 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6caf52d00a14c5bbeb443c5ee16a521" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44921 } }
I20260926 01:03:35.542311 24011 leader_election.cc:302] T 00000000000000000000000000000000 P f6caf52d00a14c5bbeb443c5ee16a521 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6caf52d00a14c5bbeb443c5ee16a521; no voters:
I20260926 01:03:35.542347 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 23942
I20260926 01:03:35.542478 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384372073861-3372-0/minicluster-data/master-0/wal/instance
I20260926 01:03:35.542492 24011 leader_election.cc:288] T 00000000000000000000000000000000 P f6caf52d00a14c5bbeb443c5ee16a521 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:03:35.542675 3372 external_mini_cluster.cc:1428] Setting key 6745cfc128d9b991865cf3d3d94ee8b9
I20260926 01:03:35.542676 24015 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6caf52d00a14c5bbeb443c5ee16a521 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:03:35.542752 24011 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6caf52d00a14c5bbeb443c5ee16a521 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:03:35.542945 24015 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6caf52d00a14c5bbeb443c5ee16a521 [term 1 LEADER]: Becoming Leader. State: Replica: f6caf52d00a14c5bbeb443c5ee16a521, State: Running, Role: LEADER
I20260926 01:03:35.543093 24015 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6caf52d00a14c5bbeb443c5ee16a521 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6caf52d00a14c5bbeb443c5ee16a521" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44921 } }
I20260926 01:03:35.543649 24015 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6caf52d00a14c5bbeb443c5ee16a521 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6caf52d00a14c5bbeb443c5ee16a521. Latest consensus state: current_term: 1 leader_uuid: "f6caf52d00a14c5bbeb443c5ee16a521" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6caf52d00a14c5bbeb443c5ee16a521" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44921 } } }
I20260926 01:03:35.543658 24014 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6caf52d00a14c5bbeb443c5ee16a521 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6caf52d00a14c5bbeb443c5ee16a521" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6caf52d00a14c5bbeb443c5ee16a521" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44921 } } }
I20260926 01:03:35.543787 24015 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6caf52d00a14c5bbeb443c5ee16a521 [sys.catalog]: This master's current role is: LEADER
I20260926 01:03:35.543793 24014 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6caf52d00a14c5bbeb443c5ee16a521 [sys.catalog]: This master's current role is: LEADER
W20260926 01:03:35.544803 24031 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f6caf52d00a14c5bbeb443c5ee16a521: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:03:35.544865 24031 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:03:35.544981 24028 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:03:35.545145 24028 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:03:35.546700 24028 catalog_manager.cc:1347] Generated new cluster ID: d4a47baec4424e25840d381635b6dc73
I20260926 01:03:35.546751 24028 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 01:03:35 dist-test-slave-7v10 krb5kdc[23929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 01:03:35.554464 24023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:35.543360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36197 (local address 127.0.0.1:44921)
0926 01:03:35.543816 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:35.543828 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:35.543864 (+ 36us) server_negotiation.cc:407] Connection header received
0926 01:03:35.543995 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:35.544003 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:35.544064 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:35.544270 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:35.545149 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:35.546401 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:35.547514 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:35.548111 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:35.548212 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:35.550968 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:35.550971 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:35.550972 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:03:35.550983 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:35.552902 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:35.553290 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:35.553293 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:35.553293 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:35.553400 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:35.553603 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:35.553604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:35.553604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:35.553878 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:35.554018 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:35.554139 (+ 121us) server_negotiation.cc:299] Negotiation successful
0926 01:03:35.554268 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":104,"threads_started":1}
I20260926 01:03:35.555392 3372 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:03:35.558213 24028 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:03:35.558737 24028 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:03:35.563102 24028 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6caf52d00a14c5bbeb443c5ee16a521: Generated new TSK 0
I20260926 01:03:35.563287 24028 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:03:35.563997 24023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:35.556933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58153 (local address 127.0.0.1:44921)
0926 01:03:35.556970 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:35.556972 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:35.556992 (+ 20us) server_negotiation.cc:407] Connection header received
0926 01:03:35.557048 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:35.557049 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:35.557059 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:35.557113 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:35.557718 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:35.558414 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:35.559455 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:35.559751 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:35.559814 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:35.561101 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:35.561102 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:35.561103 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:03:35.561109 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:35.562633 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:35.563131 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:35.563132 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:35.563132 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:35.563190 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:35.563425 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:35.563426 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:35.563426 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:35.563538 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:35.563590 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:35.563789 (+ 199us) server_negotiation.cc:299] Negotiation successful
0926 01:03:35.563880 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 01:03:35.565945 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 23942
2026-09-26T01:03:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1297 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384372073861-3372-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:03:35 dist-test-slave-7v10 krb5kdc[24042](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:03:35 dist-test-slave-7v10 krb5kdc[24042](info): set up 2 sockets
Sep 26 01:03:35 dist-test-slave-7v10 krb5kdc[24042](info): commencing operation
krb5kdc: starting...
W20260926 01:03:36.598038 3372 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Sep 26 01:03:36 dist-test-slave-7v10 krb5kdc[24042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:03:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:03:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384372073861-3372-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384372073861-3372-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 01:03:36.712110 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384372073861-3372-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384372073861-3372-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384372073861-3372-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46345
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384372073861-3372-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46345
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:03:36.791474 24055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:03:36.791666 24055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:03:36.791693 24055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:03:36.792785 24055 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:03:36.792821 24055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:03:36.792835 24055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:03:36.792848 24055 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:03:36.792860 24055 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:03:36.794180 24055 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384372073861-3372-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384372073861-3372-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46345
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384372073861-3372-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384372073861-3372-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:03:36.794348 24055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:03:36.794497 24055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:03:36.799141 24063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:36.799135 24061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:36.799135 24060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:03:36.799177 24055 server_base.cc:1034] running on GCE node
I20260926 01:03:36.799623 24055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:03:36.799902 24055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:03:36.801065 24055 hybrid_clock.cc:648] HybridClock initialized: now 1790384616801061 us; error 37 us; skew 500 ppm
Sep 26 01:03:36 dist-test-slave-7v10 krb5kdc[24042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384616, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:03:36.803535 24055 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 01:03:36.803958 24055 webserver.cc:460] Webserver started at http://127.0.0.1:43657/ using document root <none> and password file <none>
I20260926 01:03:36.804136 24055 fs_manager.cc:359] Metadata directory not provided
I20260926 01:03:36.804174 24055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:03:36.804267 24055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:03:36.805217 24055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384372073861-3372-0/minicluster-data/master-0/data/instance:
uuid: "1d376a429f004fe1a3b2b7fca82204e6"
format_stamp: "Formatted at 2026-09-26 01:03:36 on dist-test-slave-7v10"
server_key: "3471a075188cd28c0bd856bbf952f7f8"
server_key_iv: "8fc1c2c8312583ed56f49f222c7df340"
server_key_version: "encryptionkey@0"
I20260926 01:03:36.805503 24055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384372073861-3372-0/minicluster-data/master-0/wal/instance:
uuid: "1d376a429f004fe1a3b2b7fca82204e6"
format_stamp: "Formatted at 2026-09-26 01:03:36 on dist-test-slave-7v10"
server_key: "3471a075188cd28c0bd856bbf952f7f8"
server_key_iv: "8fc1c2c8312583ed56f49f222c7df340"
server_key_version: "encryptionkey@0"
I20260926 01:03:36.806697 24055 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 01:03:36.807416 24070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:36.807602 24055 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:36.807721 24055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384372073861-3372-0/minicluster-data/master-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384372073861-3372-0/minicluster-data/master-0/wal
uuid: "1d376a429f004fe1a3b2b7fca82204e6"
format_stamp: "Formatted at 2026-09-26 01:03:36 on dist-test-slave-7v10"
server_key: "3471a075188cd28c0bd856bbf952f7f8"
server_key_iv: "8fc1c2c8312583ed56f49f222c7df340"
server_key_version: "encryptionkey@0"
I20260926 01:03:36.807778 24055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384372073861-3372-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384372073861-3372-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384372073861-3372-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:03:36.825362 24055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:03:36.828572 24055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:03:36.828790 24055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:03:36.832940 24055 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46345
I20260926 01:03:36.832993 24123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46345 every 8 connection(s)
I20260926 01:03:36.833364 24055 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
I20260926 01:03:36.834328 24124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:03:36.836696 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 24055
I20260926 01:03:36.836759 24124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d376a429f004fe1a3b2b7fca82204e6: Bootstrap starting.
I20260926 01:03:36.836818 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384372073861-3372-0/minicluster-data/master-0/wal/instance
I20260926 01:03:36.837023 3372 external_mini_cluster.cc:1428] Setting key 1e5b8a5f32a6f8a621f27c91d378ddd2
I20260926 01:03:36.837448 24124 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d376a429f004fe1a3b2b7fca82204e6: Neither blocks nor log segments found. Creating new log.
I20260926 01:03:36.837829 24124 log.cc:824] T 00000000000000000000000000000000 P 1d376a429f004fe1a3b2b7fca82204e6: Log is configured to *not* fsync() on all Append() calls
I20260926 01:03:36.838567 24124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d376a429f004fe1a3b2b7fca82204e6: No bootstrap required, opened a new log
I20260926 01:03:36.840708 24124 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d376a429f004fe1a3b2b7fca82204e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d376a429f004fe1a3b2b7fca82204e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46345 } }
I20260926 01:03:36.840873 24124 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d376a429f004fe1a3b2b7fca82204e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:03:36.840914 24124 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d376a429f004fe1a3b2b7fca82204e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d376a429f004fe1a3b2b7fca82204e6, State: Initialized, Role: FOLLOWER
I20260926 01:03:36.841037 24124 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d376a429f004fe1a3b2b7fca82204e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d376a429f004fe1a3b2b7fca82204e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46345 } }
I20260926 01:03:36.841111 24124 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d376a429f004fe1a3b2b7fca82204e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:03:36.841154 24124 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d376a429f004fe1a3b2b7fca82204e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:03:36.841202 24124 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d376a429f004fe1a3b2b7fca82204e6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:03:36.841840 24124 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d376a429f004fe1a3b2b7fca82204e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d376a429f004fe1a3b2b7fca82204e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46345 } }
I20260926 01:03:36.841960 24124 leader_election.cc:302] T 00000000000000000000000000000000 P 1d376a429f004fe1a3b2b7fca82204e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d376a429f004fe1a3b2b7fca82204e6; no voters:
I20260926 01:03:36.842124 24124 leader_election.cc:288] T 00000000000000000000000000000000 P 1d376a429f004fe1a3b2b7fca82204e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:03:36.842159 24129 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d376a429f004fe1a3b2b7fca82204e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:03:36.842370 24129 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d376a429f004fe1a3b2b7fca82204e6 [term 1 LEADER]: Becoming Leader. State: Replica: 1d376a429f004fe1a3b2b7fca82204e6, State: Running, Role: LEADER
I20260926 01:03:36.842433 24124 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d376a429f004fe1a3b2b7fca82204e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:03:36.842533 24129 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d376a429f004fe1a3b2b7fca82204e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d376a429f004fe1a3b2b7fca82204e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46345 } }
I20260926 01:03:36.842901 24129 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d376a429f004fe1a3b2b7fca82204e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d376a429f004fe1a3b2b7fca82204e6. Latest consensus state: current_term: 1 leader_uuid: "1d376a429f004fe1a3b2b7fca82204e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d376a429f004fe1a3b2b7fca82204e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46345 } } }
I20260926 01:03:36.843101 24129 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d376a429f004fe1a3b2b7fca82204e6 [sys.catalog]: This master's current role is: LEADER
I20260926 01:03:36.843101 24127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:36.837550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50343 (local address 127.0.0.1:46345)
0926 01:03:36.837743 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:36.837750 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:36.837777 (+ 27us) server_negotiation.cc:407] Connection header received
0926 01:03:36.838363 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:36.838371 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:36.838420 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:36.838534 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:36.838775 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:36.839698 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:36.840894 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:36.841411 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:36.841487 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:36.842237 (+ 750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:36.842241 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:36.842241 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:03:36.842446 (+ 205us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:36.842481 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:03:36.842495 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:36.842790 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:36.842822 (+ 32us) server_negotiation.cc:299] Negotiation successful
0926 01:03:36.842909 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":97,"threads_started":1}
I20260926 01:03:36.843044 24130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d376a429f004fe1a3b2b7fca82204e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d376a429f004fe1a3b2b7fca82204e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d376a429f004fe1a3b2b7fca82204e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46345 } } }
I20260926 01:03:36.843492 24130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d376a429f004fe1a3b2b7fca82204e6 [sys.catalog]: This master's current role is: LEADER
W20260926 01:03:36.844087 24145 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1d376a429f004fe1a3b2b7fca82204e6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:03:36.844135 24145 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:03:36.844116 3372 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:03:36.844190 24144 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:03:36.844326 24144 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:03:36.845592 24144 catalog_manager.cc:1347] Generated new cluster ID: 3aa8feeacc674c8d8f934bf84c6ddc83
I20260926 01:03:36.845660 24144 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:03:36.849081 24127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:36.845400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41889 (local address 127.0.0.1:46345)
0926 01:03:36.845435 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:36.845437 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:36.845717 (+ 280us) server_negotiation.cc:407] Connection header received
0926 01:03:36.845807 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:36.845808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:36.845817 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:36.845844 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:36.846021 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:36.846694 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:36.847773 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:36.848076 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:36.848118 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:36.848838 (+ 720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:36.848839 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:36.848839 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:03:36.848843 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:36.848855 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:03:36.848860 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:36.848887 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:36.848990 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 01:03:36.849011 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 01:03:36.855994 24144 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:03:36.856453 24144 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:03:36.862098 24144 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d376a429f004fe1a3b2b7fca82204e6: Generated new TSK 0
I20260926 01:03:36.862262 24144 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:03:36.867448 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 24055
2026-09-26T01:03:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1303 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384372073861-3372-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:03:36 dist-test-slave-7v10 krb5kdc[24155](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:03:36 dist-test-slave-7v10 krb5kdc[24155](info): set up 2 sockets
Sep 26 01:03:36 dist-test-slave-7v10 krb5kdc[24155](info): commencing operation
krb5kdc: starting...
W20260926 01:03:37.898015 3372 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Sep 26 01:03:37 dist-test-slave-7v10 krb5kdc[24155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:03:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:03:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384372073861-3372-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384372073861-3372-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 01:03:38.012297 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384372073861-3372-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384372073861-3372-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384372073861-3372-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37637
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384372073861-3372-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37637
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:03:38.092458 24168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:03:38.092654 24168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:03:38.092684 24168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:03:38.093847 24168 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:03:38.093888 24168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:03:38.093904 24168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:03:38.093915 24168 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:03:38.093928 24168 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:03:38.095381 24168 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384372073861-3372-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384372073861-3372-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37637
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384372073861-3372-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384372073861-3372-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:03:38.095614 24168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:03:38.095845 24168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:03:38.100791 24174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:38.100841 24173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:38.100764 24176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:03:38.101027 24168 server_base.cc:1034] running on GCE node
I20260926 01:03:38.101289 24168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:03:38.101653 24168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:03:38.102845 24168 hybrid_clock.cc:648] HybridClock initialized: now 1790384618102825 us; error 46 us; skew 500 ppm
Sep 26 01:03:38 dist-test-slave-7v10 krb5kdc[24155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384618, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:03:38.109659 24168 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 01:03:38.110141 24168 webserver.cc:460] Webserver started at http://127.0.0.1:37309/ using document root <none> and password file <none>
I20260926 01:03:38.110352 24168 fs_manager.cc:359] Metadata directory not provided
I20260926 01:03:38.110404 24168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:03:38.110524 24168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:03:38.111778 24168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384372073861-3372-0/minicluster-data/master-0/data/instance:
uuid: "a07e6d7593fa4ac9aaa238c790650b72"
format_stamp: "Formatted at 2026-09-26 01:03:38 on dist-test-slave-7v10"
server_key: "525efb1805aef39c38022a7ee3686ec5"
server_key_iv: "e98bc73c43f06e04f636891c04a698d2"
server_key_version: "encryptionkey@0"
I20260926 01:03:38.112134 24168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384372073861-3372-0/minicluster-data/master-0/wal/instance:
uuid: "a07e6d7593fa4ac9aaa238c790650b72"
format_stamp: "Formatted at 2026-09-26 01:03:38 on dist-test-slave-7v10"
server_key: "525efb1805aef39c38022a7ee3686ec5"
server_key_iv: "e98bc73c43f06e04f636891c04a698d2"
server_key_version: "encryptionkey@0"
I20260926 01:03:38.113667 24168 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 01:03:38.114377 24183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:38.114701 24168 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 01:03:38.114799 24168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384372073861-3372-0/minicluster-data/master-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384372073861-3372-0/minicluster-data/master-0/wal
uuid: "a07e6d7593fa4ac9aaa238c790650b72"
format_stamp: "Formatted at 2026-09-26 01:03:38 on dist-test-slave-7v10"
server_key: "525efb1805aef39c38022a7ee3686ec5"
server_key_iv: "e98bc73c43f06e04f636891c04a698d2"
server_key_version: "encryptionkey@0"
I20260926 01:03:38.114857 24168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384372073861-3372-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384372073861-3372-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384372073861-3372-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:03:38.118103 24168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:03:38.120461 24168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:03:38.120617 24168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:03:38.124331 24168 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37637
I20260926 01:03:38.124366 24236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37637 every 8 connection(s)
I20260926 01:03:38.124712 24168 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
I20260926 01:03:38.125655 24237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:03:38.127429 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 24168
I20260926 01:03:38.127547 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384372073861-3372-0/minicluster-data/master-0/wal/instance
I20260926 01:03:38.127765 3372 external_mini_cluster.cc:1428] Setting key 7874d1322f84d9b612280054c94244ef
I20260926 01:03:38.128362 24237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a07e6d7593fa4ac9aaa238c790650b72: Bootstrap starting.
I20260926 01:03:38.129051 24237 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a07e6d7593fa4ac9aaa238c790650b72: Neither blocks nor log segments found. Creating new log.
I20260926 01:03:38.129382 24237 log.cc:824] T 00000000000000000000000000000000 P a07e6d7593fa4ac9aaa238c790650b72: Log is configured to *not* fsync() on all Append() calls
I20260926 01:03:38.130226 24237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a07e6d7593fa4ac9aaa238c790650b72: No bootstrap required, opened a new log
I20260926 01:03:38.130630 24240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:38.128276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39575 (local address 127.0.0.1:37637)
0926 01:03:38.128530 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:38.128539 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:38.128567 (+ 28us) server_negotiation.cc:407] Connection header received
0926 01:03:38.129196 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:38.129207 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:38.129249 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:38.129410 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:38.130181 (+ 771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:38.130185 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:38.130185 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:03:38.130189 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:38.130232 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:03:38.130246 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:38.130339 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:38.130382 (+ 43us) server_negotiation.cc:299] Negotiation successful
0926 01:03:38.130420 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":69,"threads_started":1}
I20260926 01:03:38.131652 3372 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:03:38.132637 24237 raft_consensus.cc:348] T 00000000000000000000000000000000 P a07e6d7593fa4ac9aaa238c790650b72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07e6d7593fa4ac9aaa238c790650b72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37637 } }
I20260926 01:03:38.132833 24237 raft_consensus.cc:374] T 00000000000000000000000000000000 P a07e6d7593fa4ac9aaa238c790650b72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:03:38.132907 24237 raft_consensus.cc:729] T 00000000000000000000000000000000 P a07e6d7593fa4ac9aaa238c790650b72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a07e6d7593fa4ac9aaa238c790650b72, State: Initialized, Role: FOLLOWER
I20260926 01:03:38.133050 24237 consensus_queue.cc:260] T 00000000000000000000000000000000 P a07e6d7593fa4ac9aaa238c790650b72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07e6d7593fa4ac9aaa238c790650b72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37637 } }
I20260926 01:03:38.133167 24237 raft_consensus.cc:388] T 00000000000000000000000000000000 P a07e6d7593fa4ac9aaa238c790650b72 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:03:38.133208 24237 raft_consensus.cc:482] T 00000000000000000000000000000000 P a07e6d7593fa4ac9aaa238c790650b72 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:03:38.133361 24237 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a07e6d7593fa4ac9aaa238c790650b72 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:03:38.134156 24237 raft_consensus.cc:504] T 00000000000000000000000000000000 P a07e6d7593fa4ac9aaa238c790650b72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07e6d7593fa4ac9aaa238c790650b72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37637 } }
I20260926 01:03:38.134277 24237 leader_election.cc:302] T 00000000000000000000000000000000 P a07e6d7593fa4ac9aaa238c790650b72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a07e6d7593fa4ac9aaa238c790650b72; no voters:
I20260926 01:03:38.134395 24240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:38.133334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36901 (local address 127.0.0.1:37637)
0926 01:03:38.133375 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:38.133377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:38.133485 (+ 108us) server_negotiation.cc:407] Connection header received
0926 01:03:38.133541 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:38.133542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:38.133547 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:38.133624 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:38.134143 (+ 519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:38.134144 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:38.134145 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:03:38.134150 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:38.134165 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:03:38.134169 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:38.134230 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:38.134325 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 01:03:38.134339 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 01:03:38.134476 24237 leader_election.cc:288] T 00000000000000000000000000000000 P a07e6d7593fa4ac9aaa238c790650b72 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:03:38.134665 24248 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a07e6d7593fa4ac9aaa238c790650b72 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:03:38.134763 24237 sys_catalog.cc:564] T 00000000000000000000000000000000 P a07e6d7593fa4ac9aaa238c790650b72 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:03:38.134794 24248 raft_consensus.cc:686] T 00000000000000000000000000000000 P a07e6d7593fa4ac9aaa238c790650b72 [term 1 LEADER]: Becoming Leader. State: Replica: a07e6d7593fa4ac9aaa238c790650b72, State: Running, Role: LEADER
I20260926 01:03:38.134902 24248 consensus_queue.cc:237] T 00000000000000000000000000000000 P a07e6d7593fa4ac9aaa238c790650b72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07e6d7593fa4ac9aaa238c790650b72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37637 } }
I20260926 01:03:38.135251 24249 sys_catalog.cc:455] T 00000000000000000000000000000000 P a07e6d7593fa4ac9aaa238c790650b72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a07e6d7593fa4ac9aaa238c790650b72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07e6d7593fa4ac9aaa238c790650b72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37637 } } }
I20260926 01:03:38.135413 24249 sys_catalog.cc:458] T 00000000000000000000000000000000 P a07e6d7593fa4ac9aaa238c790650b72 [sys.catalog]: This master's current role is: LEADER
I20260926 01:03:38.135244 24248 sys_catalog.cc:455] T 00000000000000000000000000000000 P a07e6d7593fa4ac9aaa238c790650b72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a07e6d7593fa4ac9aaa238c790650b72. Latest consensus state: current_term: 1 leader_uuid: "a07e6d7593fa4ac9aaa238c790650b72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07e6d7593fa4ac9aaa238c790650b72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37637 } } }
I20260926 01:03:38.135866 24248 sys_catalog.cc:458] T 00000000000000000000000000000000 P a07e6d7593fa4ac9aaa238c790650b72 [sys.catalog]: This master's current role is: LEADER
I20260926 01:03:38.136029 24256 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:03:38.136579 24256 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:03:38.138026 24256 catalog_manager.cc:1347] Generated new cluster ID: 468298793b9b4eebbc58536be8d39521
I20260926 01:03:38.138103 24256 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:03:38.164013 24256 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:03:38.164232 24256 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:03:38.177294 24256 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a07e6d7593fa4ac9aaa238c790650b72: Generated new TSK 0
I20260926 01:03:38.177506 24256 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:03:38.184101 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 24168
2026-09-26T01:03:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1314 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384372073861-3372-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:03:38 dist-test-slave-7v10 krb5kdc[24268](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:03:38 dist-test-slave-7v10 krb5kdc[24268](info): set up 2 sockets
Sep 26 01:03:38 dist-test-slave-7v10 krb5kdc[24268](info): commencing operation
krb5kdc: starting...
W20260926 01:03:39.301934 3372 mini_kdc.cc:121] Time spent starting KDC: real 1.100s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Sep 26 01:03:39 dist-test-slave-7v10 krb5kdc[24268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:03:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:03:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384372073861-3372-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384372073861-3372-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 01:03:39.417284 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384372073861-3372-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384372073861-3372-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384372073861-3372-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43075
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384372073861-3372-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43075
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:03:39.496912 24281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:03:39.497105 24281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:03:39.497128 24281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:03:39.498273 24281 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:03:39.498315 24281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:03:39.498330 24281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:03:39.498343 24281 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:03:39.498399 24281 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:03:39.499869 24281 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384372073861-3372-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384372073861-3372-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43075
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384372073861-3372-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384372073861-3372-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:03:39.500106 24281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:03:39.500332 24281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:03:39.505168 24289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:39.505203 24287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:39.505203 24286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:03:39.505323 24281 server_base.cc:1034] running on GCE node
I20260926 01:03:39.505642 24281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:03:39.505877 24281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:03:39.507071 24281 hybrid_clock.cc:648] HybridClock initialized: now 1790384619507045 us; error 30 us; skew 500 ppm
Sep 26 01:03:39 dist-test-slave-7v10 krb5kdc[24268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:03:39.509483 24281 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 01:03:39.509850 24281 webserver.cc:460] Webserver started at http://127.0.0.1:43939/ using document root <none> and password file <none>
I20260926 01:03:39.510051 24281 fs_manager.cc:359] Metadata directory not provided
I20260926 01:03:39.510097 24281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:03:39.510182 24281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:03:39.511212 24281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384372073861-3372-0/minicluster-data/master-0/data/instance:
uuid: "0cb0d3cd4c184fb4b1d81933706b6265"
format_stamp: "Formatted at 2026-09-26 01:03:39 on dist-test-slave-7v10"
server_key: "57444a4ab90ab73fcafca912ef40f08d"
server_key_iv: "6ab323b817fe02d40925e9fd6686cb2d"
server_key_version: "encryptionkey@0"
I20260926 01:03:39.511579 24281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384372073861-3372-0/minicluster-data/master-0/wal/instance:
uuid: "0cb0d3cd4c184fb4b1d81933706b6265"
format_stamp: "Formatted at 2026-09-26 01:03:39 on dist-test-slave-7v10"
server_key: "57444a4ab90ab73fcafca912ef40f08d"
server_key_iv: "6ab323b817fe02d40925e9fd6686cb2d"
server_key_version: "encryptionkey@0"
I20260926 01:03:39.513005 24281 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:03:39.513859 24296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:39.514081 24281 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:03:39.514155 24281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384372073861-3372-0/minicluster-data/master-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384372073861-3372-0/minicluster-data/master-0/wal
uuid: "0cb0d3cd4c184fb4b1d81933706b6265"
format_stamp: "Formatted at 2026-09-26 01:03:39 on dist-test-slave-7v10"
server_key: "57444a4ab90ab73fcafca912ef40f08d"
server_key_iv: "6ab323b817fe02d40925e9fd6686cb2d"
server_key_version: "encryptionkey@0"
I20260926 01:03:39.514250 24281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384372073861-3372-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384372073861-3372-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384372073861-3372-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:03:39.524912 24281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:03:39.526788 24281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:03:39.526959 24281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:03:39.530639 24281 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43075
I20260926 01:03:39.530740 24348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43075 every 8 connection(s)
I20260926 01:03:39.531072 24281 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
I20260926 01:03:39.531934 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 24281
I20260926 01:03:39.532083 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384372073861-3372-0/minicluster-data/master-0/wal/instance
I20260926 01:03:39.532354 3372 external_mini_cluster.cc:1428] Setting key 7d6e606093209d15e0d68338c56adaa7
I20260926 01:03:39.532370 24349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:03:39.535549 24349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cb0d3cd4c184fb4b1d81933706b6265: Bootstrap starting.
I20260926 01:03:39.536470 24349 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0cb0d3cd4c184fb4b1d81933706b6265: Neither blocks nor log segments found. Creating new log.
I20260926 01:03:39.536866 24349 log.cc:824] T 00000000000000000000000000000000 P 0cb0d3cd4c184fb4b1d81933706b6265: Log is configured to *not* fsync() on all Append() calls
I20260926 01:03:39.537710 24349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cb0d3cd4c184fb4b1d81933706b6265: No bootstrap required, opened a new log
Sep 26 01:03:39 dist-test-slave-7v10 krb5kdc[24268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 01:03:39.540318 24349 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0cb0d3cd4c184fb4b1d81933706b6265 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cb0d3cd4c184fb4b1d81933706b6265" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43075 } }
I20260926 01:03:39.540510 24349 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0cb0d3cd4c184fb4b1d81933706b6265 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:03:39.540593 24349 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0cb0d3cd4c184fb4b1d81933706b6265 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cb0d3cd4c184fb4b1d81933706b6265, State: Initialized, Role: FOLLOWER
I20260926 01:03:39.540755 24349 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0cb0d3cd4c184fb4b1d81933706b6265 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cb0d3cd4c184fb4b1d81933706b6265" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43075 } }
I20260926 01:03:39.540925 24349 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0cb0d3cd4c184fb4b1d81933706b6265 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:03:39.540985 24349 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0cb0d3cd4c184fb4b1d81933706b6265 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:03:39.541064 24349 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0cb0d3cd4c184fb4b1d81933706b6265 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:03:39.541898 24349 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0cb0d3cd4c184fb4b1d81933706b6265 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cb0d3cd4c184fb4b1d81933706b6265" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43075 } }
I20260926 01:03:39.542037 24349 leader_election.cc:302] T 00000000000000000000000000000000 P 0cb0d3cd4c184fb4b1d81933706b6265 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0cb0d3cd4c184fb4b1d81933706b6265; no voters:
I20260926 01:03:39.542229 24349 leader_election.cc:288] T 00000000000000000000000000000000 P 0cb0d3cd4c184fb4b1d81933706b6265 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:03:39.542265 24354 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0cb0d3cd4c184fb4b1d81933706b6265 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:03:39.542461 24354 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0cb0d3cd4c184fb4b1d81933706b6265 [term 1 LEADER]: Becoming Leader. State: Replica: 0cb0d3cd4c184fb4b1d81933706b6265, State: Running, Role: LEADER
I20260926 01:03:39.542522 24349 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0cb0d3cd4c184fb4b1d81933706b6265 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:03:39.542587 24354 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0cb0d3cd4c184fb4b1d81933706b6265 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cb0d3cd4c184fb4b1d81933706b6265" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43075 } }
I20260926 01:03:39.542975 24354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cb0d3cd4c184fb4b1d81933706b6265 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0cb0d3cd4c184fb4b1d81933706b6265. Latest consensus state: current_term: 1 leader_uuid: "0cb0d3cd4c184fb4b1d81933706b6265" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cb0d3cd4c184fb4b1d81933706b6265" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43075 } } }
I20260926 01:03:39.543208 24354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cb0d3cd4c184fb4b1d81933706b6265 [sys.catalog]: This master's current role is: LEADER
I20260926 01:03:39.543125 24356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cb0d3cd4c184fb4b1d81933706b6265 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0cb0d3cd4c184fb4b1d81933706b6265" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cb0d3cd4c184fb4b1d81933706b6265" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43075 } } }
I20260926 01:03:39.543628 24356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cb0d3cd4c184fb4b1d81933706b6265 [sys.catalog]: This master's current role is: LEADER
I20260926 01:03:39.543632 24362 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:03:39.544178 24362 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:03:39.545009 24352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:39.532895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.110:58261 (local address 127.0.0.1:43075)
0926 01:03:39.533089 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:39.533096 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:39.533119 (+ 23us) server_negotiation.cc:407] Connection header received
0926 01:03:39.533497 (+ 378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:39.533503 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:39.533546 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:39.533776 (+ 230us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:39.534440 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:39.535361 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:39.536455 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:39.537192 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:39.537302 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:39.540009 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:39.540014 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:39.540014 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:03:39.540027 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:39.542377 (+ 2350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:39.543031 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:39.543034 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:39.543035 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:39.543166 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:39.543545 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:39.543545 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:39.543546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:39.543900 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:39.544534 (+ 634us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:39.544636 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 01:03:39.544758 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":90,"threads_started":1}
I20260926 01:03:39.545565 24362 catalog_manager.cc:1347] Generated new cluster ID: 9711c0f0128848c093c9011d9afa0458
I20260926 01:03:39.545619 24362 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:03:39.546151 3372 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:03:39.557485 24352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:39.551231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.110:53783 (local address 127.0.0.1:43075)
0926 01:03:39.551271 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:39.551273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:39.551431 (+ 158us) server_negotiation.cc:407] Connection header received
0926 01:03:39.551505 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:39.551506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:39.551516 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:39.551622 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:39.552171 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:39.552645 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:39.553346 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:39.553573 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:39.553645 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:39.554864 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:39.554865 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:39.554866 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:03:39.554902 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:39.556315 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:39.556676 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:39.556677 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:39.556677 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:39.556738 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:39.556958 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:39.556959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:39.556960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:39.557067 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:39.557176 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:39.557299 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 01:03:39.557366 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 01:03:39.567960 24362 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:03:39.568658 24362 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:03:39.574281 24362 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0cb0d3cd4c184fb4b1d81933706b6265: Generated new TSK 0
I20260926 01:03:39.574491 24362 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:03:39.579622 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 24281
W20260926 01:03:39.584113 24266 connection.cc:535] client connection to 127.0.0.1:43075 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43075): Connection reset by peer (error 104)
2026-09-26T01:03:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1395 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384372073861-3372-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:03:39 dist-test-slave-7v10 krb5kdc[24381](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:03:39 dist-test-slave-7v10 krb5kdc[24381](info): set up 2 sockets
Sep 26 01:03:39 dist-test-slave-7v10 krb5kdc[24381](info): commencing operation
krb5kdc: starting...
W20260926 01:03:40.598315 3372 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Sep 26 01:03:40 dist-test-slave-7v10 krb5kdc[24381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:03:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:03:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384372073861-3372-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384372073861-3372-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 01:03:40.715076 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384372073861-3372-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384372073861-3372-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384372073861-3372-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40817
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384372073861-3372-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40817
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:03:40.794451 24394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:03:40.794775 24394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:03:40.794818 24394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:03:40.795992 24394 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:03:40.796033 24394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:03:40.796047 24394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:03:40.796092 24394 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:03:40.796119 24394 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:03:40.797567 24394 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384372073861-3372-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384372073861-3372-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384372073861-3372-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384372073861-3372-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:03:40.797816 24394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:03:40.798028 24394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:03:40.802953 24400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:40.803016 24399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:40.802992 24402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:03:40.803094 24394 server_base.cc:1034] running on GCE node
I20260926 01:03:40.803401 24394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:03:40.803717 24394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:03:40.804876 24394 hybrid_clock.cc:648] HybridClock initialized: now 1790384620804871 us; error 41 us; skew 500 ppm
Sep 26 01:03:40 dist-test-slave-7v10 krb5kdc[24381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384620, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:03:40.807453 24394 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 01:03:40.807962 24394 webserver.cc:460] Webserver started at http://127.0.0.1:45191/ using document root <none> and password file <none>
I20260926 01:03:40.808233 24394 fs_manager.cc:359] Metadata directory not provided
I20260926 01:03:40.808310 24394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:03:40.808475 24394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:03:40.809584 24394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384372073861-3372-0/minicluster-data/master-0/data/instance:
uuid: "f65a3d54cc74482f9c48b56b5c34fd23"
format_stamp: "Formatted at 2026-09-26 01:03:40 on dist-test-slave-7v10"
server_key: "7460ea506d6600c9eaf02fb7b72ef0ec"
server_key_iv: "64f0016baa9a2d60a2da6bf3a09c09ab"
server_key_version: "encryptionkey@0"
I20260926 01:03:40.809969 24394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384372073861-3372-0/minicluster-data/master-0/wal/instance:
uuid: "f65a3d54cc74482f9c48b56b5c34fd23"
format_stamp: "Formatted at 2026-09-26 01:03:40 on dist-test-slave-7v10"
server_key: "7460ea506d6600c9eaf02fb7b72ef0ec"
server_key_iv: "64f0016baa9a2d60a2da6bf3a09c09ab"
server_key_version: "encryptionkey@0"
I20260926 01:03:40.811420 24394 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 01:03:40.812148 24409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:40.812317 24394 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 01:03:40.812423 24394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384372073861-3372-0/minicluster-data/master-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384372073861-3372-0/minicluster-data/master-0/wal
uuid: "f65a3d54cc74482f9c48b56b5c34fd23"
format_stamp: "Formatted at 2026-09-26 01:03:40 on dist-test-slave-7v10"
server_key: "7460ea506d6600c9eaf02fb7b72ef0ec"
server_key_iv: "64f0016baa9a2d60a2da6bf3a09c09ab"
server_key_version: "encryptionkey@0"
I20260926 01:03:40.812510 24394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384372073861-3372-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384372073861-3372-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384372073861-3372-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:03:40.848048 24394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:03:40.852128 24394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:03:40.852291 24394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:03:40.856242 24394 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40817
I20260926 01:03:40.856321 24461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40817 every 8 connection(s)
I20260926 01:03:40.856678 24394 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
I20260926 01:03:40.857738 24462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:03:40.859768 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 24394
I20260926 01:03:40.859900 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384372073861-3372-0/minicluster-data/master-0/wal/instance
I20260926 01:03:40.860137 3372 external_mini_cluster.cc:1428] Setting key 5e4ac07a474c2ae3c0da059d9d04dac6
I20260926 01:03:40.860630 24462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f65a3d54cc74482f9c48b56b5c34fd23: Bootstrap starting.
I20260926 01:03:40.861264 24462 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f65a3d54cc74482f9c48b56b5c34fd23: Neither blocks nor log segments found. Creating new log.
I20260926 01:03:40.861641 24462 log.cc:824] T 00000000000000000000000000000000 P f65a3d54cc74482f9c48b56b5c34fd23: Log is configured to *not* fsync() on all Append() calls
I20260926 01:03:40.862378 24462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f65a3d54cc74482f9c48b56b5c34fd23: No bootstrap required, opened a new log
I20260926 01:03:40.864836 24462 raft_consensus.cc:348] T 00000000000000000000000000000000 P f65a3d54cc74482f9c48b56b5c34fd23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65a3d54cc74482f9c48b56b5c34fd23" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40817 } }
I20260926 01:03:40.865037 24462 raft_consensus.cc:374] T 00000000000000000000000000000000 P f65a3d54cc74482f9c48b56b5c34fd23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:03:40.865087 24462 raft_consensus.cc:729] T 00000000000000000000000000000000 P f65a3d54cc74482f9c48b56b5c34fd23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65a3d54cc74482f9c48b56b5c34fd23, State: Initialized, Role: FOLLOWER
I20260926 01:03:40.865228 24462 consensus_queue.cc:260] T 00000000000000000000000000000000 P f65a3d54cc74482f9c48b56b5c34fd23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65a3d54cc74482f9c48b56b5c34fd23" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40817 } }
I20260926 01:03:40.865332 24462 raft_consensus.cc:388] T 00000000000000000000000000000000 P f65a3d54cc74482f9c48b56b5c34fd23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:03:40.865379 24462 raft_consensus.cc:482] T 00000000000000000000000000000000 P f65a3d54cc74482f9c48b56b5c34fd23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:03:40.865432 24462 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f65a3d54cc74482f9c48b56b5c34fd23 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:03:40.865875 24465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:40.860680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.110:35719 (local address 127.0.0.1:40817)
0926 01:03:40.860851 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:40.860858 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:40.860882 (+ 24us) server_negotiation.cc:407] Connection header received
0926 01:03:40.861295 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:40.861301 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:40.861351 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:40.861513 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:40.861792 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:40.862774 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:40.863921 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:40.864309 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:40.864391 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:40.865131 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:40.865135 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:40.865135 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:03:40.865333 (+ 198us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:40.865387 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:03:40.865403 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:40.865478 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:40.865636 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 01:03:40.865667 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":79,"threads_started":1}
I20260926 01:03:40.866068 24462 raft_consensus.cc:504] T 00000000000000000000000000000000 P f65a3d54cc74482f9c48b56b5c34fd23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65a3d54cc74482f9c48b56b5c34fd23" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40817 } }
I20260926 01:03:40.866182 24462 leader_election.cc:302] T 00000000000000000000000000000000 P f65a3d54cc74482f9c48b56b5c34fd23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f65a3d54cc74482f9c48b56b5c34fd23; no voters:
I20260926 01:03:40.866354 24462 leader_election.cc:288] T 00000000000000000000000000000000 P f65a3d54cc74482f9c48b56b5c34fd23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:03:40.866379 24467 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f65a3d54cc74482f9c48b56b5c34fd23 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:03:40.866593 24467 raft_consensus.cc:686] T 00000000000000000000000000000000 P f65a3d54cc74482f9c48b56b5c34fd23 [term 1 LEADER]: Becoming Leader. State: Replica: f65a3d54cc74482f9c48b56b5c34fd23, State: Running, Role: LEADER
I20260926 01:03:40.866739 24462 sys_catalog.cc:564] T 00000000000000000000000000000000 P f65a3d54cc74482f9c48b56b5c34fd23 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:03:40.866804 24467 consensus_queue.cc:237] T 00000000000000000000000000000000 P f65a3d54cc74482f9c48b56b5c34fd23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65a3d54cc74482f9c48b56b5c34fd23" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40817 } }
I20260926 01:03:40.866921 3372 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:03:40.867897 24468 sys_catalog.cc:455] T 00000000000000000000000000000000 P f65a3d54cc74482f9c48b56b5c34fd23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f65a3d54cc74482f9c48b56b5c34fd23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65a3d54cc74482f9c48b56b5c34fd23" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40817 } } }
I20260926 01:03:40.867982 24468 sys_catalog.cc:458] T 00000000000000000000000000000000 P f65a3d54cc74482f9c48b56b5c34fd23 [sys.catalog]: This master's current role is: LEADER
I20260926 01:03:40.868466 24469 sys_catalog.cc:455] T 00000000000000000000000000000000 P f65a3d54cc74482f9c48b56b5c34fd23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f65a3d54cc74482f9c48b56b5c34fd23. Latest consensus state: current_term: 1 leader_uuid: "f65a3d54cc74482f9c48b56b5c34fd23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65a3d54cc74482f9c48b56b5c34fd23" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40817 } } }
I20260926 01:03:40.868551 24469 sys_catalog.cc:458] T 00000000000000000000000000000000 P f65a3d54cc74482f9c48b56b5c34fd23 [sys.catalog]: This master's current role is: LEADER
I20260926 01:03:40.868961 24484 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:03:40.869484 24484 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:03:40.871184 24484 catalog_manager.cc:1347] Generated new cluster ID: 88a2bf4eefca4c6f867e43d7687bba4e
I20260926 01:03:40.871233 24484 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:03:40.872304 24465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:40.868829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.110:48817 (local address 127.0.0.1:40817)
0926 01:03:40.868876 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:40.868878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:40.868952 (+ 74us) server_negotiation.cc:407] Connection header received
0926 01:03:40.869022 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:40.869023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:40.869032 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:40.869077 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:40.869250 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:40.869946 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:40.870954 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:40.871254 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:40.871339 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:40.872049 (+ 710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:40.872050 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:40.872050 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:03:40.872054 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:40.872069 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:03:40.872074 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:40.872137 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:40.872222 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 01:03:40.872233 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 01:03:40.880604 24484 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:03:40.881121 24484 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:03:40.885582 24484 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f65a3d54cc74482f9c48b56b5c34fd23: Generated new TSK 0
I20260926 01:03:40.885797 24484 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:03:40.895394 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 24394
W20260926 01:03:40.900079 24379 connection.cc:535] client connection to 127.0.0.1:40817 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40817): Connection reset by peer (error 104)
2026-09-26T01:03:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1316 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384372073861-3372-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:03:40 dist-test-slave-7v10 krb5kdc[24493](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:03:40 dist-test-slave-7v10 krb5kdc[24493](info): set up 2 sockets
Sep 26 01:03:40 dist-test-slave-7v10 krb5kdc[24493](info): commencing operation
krb5kdc: starting...
W20260926 01:03:41.938848 3372 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Sep 26 01:03:42 dist-test-slave-7v10 krb5kdc[24493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:03:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:03:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384372073861-3372-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384372073861-3372-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 01:03:42.053699 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384372073861-3372-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384372073861-3372-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384372073861-3372-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41141
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384372073861-3372-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41141
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:03:42.132833 24506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:03:42.133028 24506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:03:42.133049 24506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:03:42.134212 24506 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:03:42.134250 24506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:03:42.134264 24506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:03:42.134277 24506 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:03:42.134334 24506 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:03:42.135848 24506 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384372073861-3372-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384372073861-3372-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384372073861-3372-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384372073861-3372-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:03:42.136094 24506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:03:42.136320 24506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:03:42.141075 24514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:42.141069 24512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:42.141103 24511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:03:42.141101 24506 server_base.cc:1034] running on GCE node
I20260926 01:03:42.141542 24506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:03:42.141866 24506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:03:42.143019 24506 hybrid_clock.cc:648] HybridClock initialized: now 1790384622143014 us; error 35 us; skew 500 ppm
Sep 26 01:03:42 dist-test-slave-7v10 krb5kdc[24493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384622, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:03:42.145493 24506 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 01:03:42.145933 24506 webserver.cc:460] Webserver started at http://127.0.0.1:34061/ using document root <none> and password file <none>
I20260926 01:03:42.146127 24506 fs_manager.cc:359] Metadata directory not provided
I20260926 01:03:42.146168 24506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:03:42.146268 24506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:03:42.147305 24506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384372073861-3372-0/minicluster-data/master-0/data/instance:
uuid: "ca506be42a8c4dceacb95bd26f70c830"
format_stamp: "Formatted at 2026-09-26 01:03:42 on dist-test-slave-7v10"
server_key: "3c62420937fad97d69505d5a5522dadf"
server_key_iv: "c02005693189a32861019f47bbbc488f"
server_key_version: "encryptionkey@0"
I20260926 01:03:42.147589 24506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384372073861-3372-0/minicluster-data/master-0/wal/instance:
uuid: "ca506be42a8c4dceacb95bd26f70c830"
format_stamp: "Formatted at 2026-09-26 01:03:42 on dist-test-slave-7v10"
server_key: "3c62420937fad97d69505d5a5522dadf"
server_key_iv: "c02005693189a32861019f47bbbc488f"
server_key_version: "encryptionkey@0"
I20260926 01:03:42.148792 24506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:03:42.149433 24521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:42.149605 24506 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:03:42.149693 24506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384372073861-3372-0/minicluster-data/master-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384372073861-3372-0/minicluster-data/master-0/wal
uuid: "ca506be42a8c4dceacb95bd26f70c830"
format_stamp: "Formatted at 2026-09-26 01:03:42 on dist-test-slave-7v10"
server_key: "3c62420937fad97d69505d5a5522dadf"
server_key_iv: "c02005693189a32861019f47bbbc488f"
server_key_version: "encryptionkey@0"
I20260926 01:03:42.149744 24506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384372073861-3372-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384372073861-3372-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384372073861-3372-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:03:42.152873 24506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:03:42.154632 24506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:03:42.154732 24506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:03:42.158524 24506 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41141
I20260926 01:03:42.158566 24575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41141 every 8 connection(s)
I20260926 01:03:42.158919 24506 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
I20260926 01:03:42.159862 24576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:03:42.161995 24576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca506be42a8c4dceacb95bd26f70c830: Bootstrap starting.
I20260926 01:03:42.162539 24576 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca506be42a8c4dceacb95bd26f70c830: Neither blocks nor log segments found. Creating new log.
I20260926 01:03:42.162818 24576 log.cc:824] T 00000000000000000000000000000000 P ca506be42a8c4dceacb95bd26f70c830: Log is configured to *not* fsync() on all Append() calls
I20260926 01:03:42.163478 24576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca506be42a8c4dceacb95bd26f70c830: No bootstrap required, opened a new log
I20260926 01:03:42.165006 24576 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca506be42a8c4dceacb95bd26f70c830 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca506be42a8c4dceacb95bd26f70c830" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41141 } }
I20260926 01:03:42.165167 24576 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca506be42a8c4dceacb95bd26f70c830 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:03:42.165232 24576 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca506be42a8c4dceacb95bd26f70c830 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca506be42a8c4dceacb95bd26f70c830, State: Initialized, Role: FOLLOWER
I20260926 01:03:42.165405 24576 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca506be42a8c4dceacb95bd26f70c830 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca506be42a8c4dceacb95bd26f70c830" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41141 } }
I20260926 01:03:42.165504 24576 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca506be42a8c4dceacb95bd26f70c830 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:03:42.165545 24576 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca506be42a8c4dceacb95bd26f70c830 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:03:42.165593 24576 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca506be42a8c4dceacb95bd26f70c830 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:03:42.166085 24576 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca506be42a8c4dceacb95bd26f70c830 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca506be42a8c4dceacb95bd26f70c830" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41141 } }
I20260926 01:03:42.166226 24576 leader_election.cc:302] T 00000000000000000000000000000000 P ca506be42a8c4dceacb95bd26f70c830 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca506be42a8c4dceacb95bd26f70c830; no voters:
I20260926 01:03:42.166395 24576 leader_election.cc:288] T 00000000000000000000000000000000 P ca506be42a8c4dceacb95bd26f70c830 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:03:42.166421 24579 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca506be42a8c4dceacb95bd26f70c830 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:03:42.166589 24579 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca506be42a8c4dceacb95bd26f70c830 [term 1 LEADER]: Becoming Leader. State: Replica: ca506be42a8c4dceacb95bd26f70c830, State: Running, Role: LEADER
I20260926 01:03:42.166630 24576 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca506be42a8c4dceacb95bd26f70c830 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:03:42.166719 24579 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca506be42a8c4dceacb95bd26f70c830 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca506be42a8c4dceacb95bd26f70c830" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41141 } }
I20260926 01:03:42.167094 24581 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca506be42a8c4dceacb95bd26f70c830 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca506be42a8c4dceacb95bd26f70c830. Latest consensus state: current_term: 1 leader_uuid: "ca506be42a8c4dceacb95bd26f70c830" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca506be42a8c4dceacb95bd26f70c830" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41141 } } }
I20260926 01:03:42.167229 24580 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca506be42a8c4dceacb95bd26f70c830 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca506be42a8c4dceacb95bd26f70c830" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca506be42a8c4dceacb95bd26f70c830" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41141 } } }
I20260926 01:03:42.167233 24581 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca506be42a8c4dceacb95bd26f70c830 [sys.catalog]: This master's current role is: LEADER
I20260926 01:03:42.167330 24580 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca506be42a8c4dceacb95bd26f70c830 [sys.catalog]: This master's current role is: LEADER
I20260926 01:03:42.167654 24588 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:03:42.168175 24588 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:03:42.168294 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 24506
I20260926 01:03:42.168409 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384372073861-3372-0/minicluster-data/master-0/wal/instance
I20260926 01:03:42.168606 3372 external_mini_cluster.cc:1428] Setting key 164868231dd0f357437a77707f08f0f5
I20260926 01:03:42.169806 24588 catalog_manager.cc:1347] Generated new cluster ID: e12f3fd1382947ed937d69544caee02a
I20260926 01:03:42.169863 24588 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:03:42.172395 24598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:42.169110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.110:52847 (local address 127.0.0.1:41141)
0926 01:03:42.171022 (+ 1912us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:42.171030 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:42.171062 (+ 32us) server_negotiation.cc:407] Connection header received
0926 01:03:42.171189 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:42.171196 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:42.171226 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:42.171366 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:42.171998 (+ 632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:42.172001 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:42.172002 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:03:42.172005 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:42.172036 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:03:42.172049 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:42.172116 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:42.172191 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 01:03:42.172232 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1863,"thread_start_us":65,"threads_started":1}
I20260926 01:03:42.173249 3372 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:03:42.180007 24598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:42.174615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.110:44341 (local address 127.0.0.1:41141)
0926 01:03:42.174643 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:42.174645 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:42.179076 (+ 4431us) server_negotiation.cc:407] Connection header received
0926 01:03:42.179122 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:42.179123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:42.179129 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:42.179190 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:42.179792 (+ 602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:42.179794 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:42.179794 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:03:42.179798 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:42.179813 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:03:42.179818 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:42.179872 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:42.179933 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 01:03:42.179946 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260926 01:03:42.180948 24588 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:03:42.181085 24588 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:03:42.188824 24588 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca506be42a8c4dceacb95bd26f70c830: Generated new TSK 0
I20260926 01:03:42.188983 24588 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:03:42.196200 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 24506
2026-09-26T01:03:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1300 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384372073861-3372-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:03:42 dist-test-slave-7v10 krb5kdc[24607](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:03:42 dist-test-slave-7v10 krb5kdc[24607](info): set up 2 sockets
Sep 26 01:03:42 dist-test-slave-7v10 krb5kdc[24607](info): commencing operation
krb5kdc: starting...
W20260926 01:03:43.226163 3372 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Sep 26 01:03:43 dist-test-slave-7v10 krb5kdc[24607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:03:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:03:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384372073861-3372-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384372073861-3372-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 01:03:43.342058 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384372073861-3372-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384372073861-3372-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384372073861-3372-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32813
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384372073861-3372-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:03:43.423486 24620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:03:43.423781 24620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:03:43.423816 24620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:03:43.425577 24620 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:03:43.425639 24620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:03:43.425663 24620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:03:43.425693 24620 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:03:43.425716 24620 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:03:43.428299 24620 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384372073861-3372-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384372073861-3372-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32813
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384372073861-3372-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384372073861-3372-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:03:43.428555 24620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:03:43.428822 24620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:03:43.434034 24628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:43.434219 24626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:43.434336 24625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:03:43.434922 24620 server_base.cc:1034] running on GCE node
I20260926 01:03:43.435155 24620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:03:43.435451 24620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:03:43.436631 24620 hybrid_clock.cc:648] HybridClock initialized: now 1790384623436565 us; error 76 us; skew 500 ppm
Sep 26 01:03:43 dist-test-slave-7v10 krb5kdc[24607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384623, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:03:43.439388 24620 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 01:03:43.439826 24620 webserver.cc:460] Webserver started at http://127.0.0.1:36347/ using document root <none> and password file <none>
I20260926 01:03:43.440104 24620 fs_manager.cc:359] Metadata directory not provided
I20260926 01:03:43.440167 24620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:03:43.440292 24620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:03:43.441646 24620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384372073861-3372-0/minicluster-data/master-0/data/instance:
uuid: "ef010385dac64fa89c3203a0f769a67b"
format_stamp: "Formatted at 2026-09-26 01:03:43 on dist-test-slave-7v10"
server_key: "494c44efa454108ae3ee712d8995a673"
server_key_iv: "b83adf61ffd91fba80cd245dc0a018f1"
server_key_version: "encryptionkey@0"
I20260926 01:03:43.442050 24620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384372073861-3372-0/minicluster-data/master-0/wal/instance:
uuid: "ef010385dac64fa89c3203a0f769a67b"
format_stamp: "Formatted at 2026-09-26 01:03:43 on dist-test-slave-7v10"
server_key: "494c44efa454108ae3ee712d8995a673"
server_key_iv: "b83adf61ffd91fba80cd245dc0a018f1"
server_key_version: "encryptionkey@0"
I20260926 01:03:43.443787 24620 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260926 01:03:43.444669 24635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:43.444914 24620 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:03:43.445015 24620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384372073861-3372-0/minicluster-data/master-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384372073861-3372-0/minicluster-data/master-0/wal
uuid: "ef010385dac64fa89c3203a0f769a67b"
format_stamp: "Formatted at 2026-09-26 01:03:43 on dist-test-slave-7v10"
server_key: "494c44efa454108ae3ee712d8995a673"
server_key_iv: "b83adf61ffd91fba80cd245dc0a018f1"
server_key_version: "encryptionkey@0"
I20260926 01:03:43.445096 24620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384372073861-3372-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384372073861-3372-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384372073861-3372-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:03:43.468247 24620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:03:43.471045 24620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:03:43.471158 24620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:03:43.475042 24620 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32813
I20260926 01:03:43.475224 24688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32813 every 8 connection(s)
I20260926 01:03:43.475449 24620 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
I20260926 01:03:43.476384 24689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:03:43.476815 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 24620
I20260926 01:03:43.476946 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384372073861-3372-0/minicluster-data/master-0/wal/instance
I20260926 01:03:43.477147 3372 external_mini_cluster.cc:1428] Setting key 63666ec58e7e3aa0c9c45b07a3bf8c59
I20260926 01:03:43.479530 24689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef010385dac64fa89c3203a0f769a67b: Bootstrap starting.
I20260926 01:03:43.480111 24689 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef010385dac64fa89c3203a0f769a67b: Neither blocks nor log segments found. Creating new log.
I20260926 01:03:43.480396 24689 log.cc:824] T 00000000000000000000000000000000 P ef010385dac64fa89c3203a0f769a67b: Log is configured to *not* fsync() on all Append() calls
I20260926 01:03:43.481053 24689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef010385dac64fa89c3203a0f769a67b: No bootstrap required, opened a new log
I20260926 01:03:43.482777 24689 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef010385dac64fa89c3203a0f769a67b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef010385dac64fa89c3203a0f769a67b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32813 } }
I20260926 01:03:43.482990 24689 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef010385dac64fa89c3203a0f769a67b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:03:43.483058 24689 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef010385dac64fa89c3203a0f769a67b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef010385dac64fa89c3203a0f769a67b, State: Initialized, Role: FOLLOWER
I20260926 01:03:43.483209 24689 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef010385dac64fa89c3203a0f769a67b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef010385dac64fa89c3203a0f769a67b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32813 } }
I20260926 01:03:43.483330 24689 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef010385dac64fa89c3203a0f769a67b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:03:43.483385 24689 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef010385dac64fa89c3203a0f769a67b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:03:43.483435 24689 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef010385dac64fa89c3203a0f769a67b [term 0 FOLLOWER]: Advancing to term 1
Sep 26 01:03:43 dist-test-slave-7v10 krb5kdc[24607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 01:03:43.484117 24689 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef010385dac64fa89c3203a0f769a67b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef010385dac64fa89c3203a0f769a67b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32813 } }
I20260926 01:03:43.484251 24689 leader_election.cc:302] T 00000000000000000000000000000000 P ef010385dac64fa89c3203a0f769a67b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef010385dac64fa89c3203a0f769a67b; no voters:
I20260926 01:03:43.484422 24689 leader_election.cc:288] T 00000000000000000000000000000000 P ef010385dac64fa89c3203a0f769a67b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:03:43.484465 24694 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef010385dac64fa89c3203a0f769a67b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:03:43.484618 24694 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef010385dac64fa89c3203a0f769a67b [term 1 LEADER]: Becoming Leader. State: Replica: ef010385dac64fa89c3203a0f769a67b, State: Running, Role: LEADER
I20260926 01:03:43.484680 24689 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef010385dac64fa89c3203a0f769a67b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:03:43.484732 24694 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef010385dac64fa89c3203a0f769a67b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef010385dac64fa89c3203a0f769a67b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32813 } }
I20260926 01:03:43.485069 24694 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef010385dac64fa89c3203a0f769a67b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef010385dac64fa89c3203a0f769a67b. Latest consensus state: current_term: 1 leader_uuid: "ef010385dac64fa89c3203a0f769a67b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef010385dac64fa89c3203a0f769a67b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32813 } } }
I20260926 01:03:43.485078 24695 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef010385dac64fa89c3203a0f769a67b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef010385dac64fa89c3203a0f769a67b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef010385dac64fa89c3203a0f769a67b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32813 } } }
I20260926 01:03:43.485167 24694 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef010385dac64fa89c3203a0f769a67b [sys.catalog]: This master's current role is: LEADER
I20260926 01:03:43.485183 24695 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef010385dac64fa89c3203a0f769a67b [sys.catalog]: This master's current role is: LEADER
W20260926 01:03:43.486522 24710 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ef010385dac64fa89c3203a0f769a67b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:03:43.486583 24710 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:03:43.486685 24700 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:03:43.486920 24700 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:03:43.488727 24700 catalog_manager.cc:1347] Generated new cluster ID: 4acc807ead3e44759d1a458ddf4d203c
I20260926 01:03:43.488765 24700 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:03:43.489506 24692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:43.477709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.110:40943 (local address 127.0.0.1:32813)
0926 01:03:43.477855 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:43.477865 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:43.477898 (+ 33us) server_negotiation.cc:407] Connection header received
0926 01:03:43.478297 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:43.478304 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:43.478352 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:43.478469 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:43.479296 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:43.480209 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:43.481016 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:43.481476 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:43.481593 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:43.484212 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:43.484217 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:43.484217 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:03:43.484230 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:43.487429 (+ 3199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:43.487907 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:43.487911 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:43.487911 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:43.488011 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:43.488281 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:43.488283 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:43.488283 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:43.488571 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:43.488682 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:43.489200 (+ 518us) server_negotiation.cc:299] Negotiation successful
0926 01:03:43.489307 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":72,"threads_started":1}
I20260926 01:03:43.490585 3372 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:03:43.499581 24700 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:03:43.500259 24700 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:03:43.505206 24692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:43.495159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.110:47813 (local address 127.0.0.1:32813)
0926 01:03:43.495199 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:43.495201 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:43.495257 (+ 56us) server_negotiation.cc:407] Connection header received
0926 01:03:43.495328 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:43.495329 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:43.495339 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:43.495417 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:43.495962 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:43.496698 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:43.497581 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:43.497888 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:43.497953 (+ 65us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:43.499348 (+ 1395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:43.499349 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:43.499350 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:03:43.499356 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:43.504015 (+ 4659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:43.504365 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:43.504383 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:43.504383 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:43.504438 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:43.504661 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:43.504662 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:43.504663 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:43.504793 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:43.504919 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:43.505044 (+ 125us) server_negotiation.cc:299] Negotiation successful
0926 01:03:43.505104 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 01:03:43.528844 24700 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef010385dac64fa89c3203a0f769a67b: Generated new TSK 0
I20260926 01:03:43.529039 24700 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:03:43.537632 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 24620
W20260926 01:03:43.542204 24605 connection.cc:535] client connection to 127.0.0.1:32813 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32813): Connection reset by peer (error 104)
2026-09-26T01:03:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1341 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384372073861-3372-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:03:43 dist-test-slave-7v10 krb5kdc[24720](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:03:43 dist-test-slave-7v10 krb5kdc[24720](info): set up 2 sockets
Sep 26 01:03:43 dist-test-slave-7v10 krb5kdc[24720](info): commencing operation
krb5kdc: starting...
W20260926 01:03:44.569959 3372 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Sep 26 01:03:44 dist-test-slave-7v10 krb5kdc[24720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:03:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:03:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384372073861-3372-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384372073861-3372-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 01:03:44.681617 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384372073861-3372-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384372073861-3372-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384372073861-3372-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43135
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384372073861-3372-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43135
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:03:44.763517 24734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:03:44.763782 24734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:03:44.763820 24734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:03:44.764981 24734 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:03:44.765048 24734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:03:44.765072 24734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:03:44.765100 24734 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:03:44.765129 24734 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:03:44.766590 24734 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384372073861-3372-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384372073861-3372-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43135
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384372073861-3372-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384372073861-3372-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:03:44.766899 24734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:03:44.767136 24734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:03:44.771953 24742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:44.771960 24739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:44.771973 24740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:03:44.772190 24734 server_base.cc:1034] running on GCE node
I20260926 01:03:44.772411 24734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:03:44.772645 24734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:03:44.773823 24734 hybrid_clock.cc:648] HybridClock initialized: now 1790384624773825 us; error 42 us; skew 500 ppm
Sep 26 01:03:44 dist-test-slave-7v10 krb5kdc[24720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:03:44.776377 24734 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 01:03:44.776805 24734 webserver.cc:460] Webserver started at http://127.0.0.1:41505/ using document root <none> and password file <none>
I20260926 01:03:44.777045 24734 fs_manager.cc:359] Metadata directory not provided
I20260926 01:03:44.777112 24734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:03:44.777268 24734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:03:44.778296 24734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384372073861-3372-0/minicluster-data/master-0/data/instance:
uuid: "e75d92b5931948fa86c0e8df498b908b"
format_stamp: "Formatted at 2026-09-26 01:03:44 on dist-test-slave-7v10"
server_key: "9794a8f3a0c31f265ec8ed5df856e106"
server_key_iv: "93dbc09d9841436782cff3826b0b9d74"
server_key_version: "encryptionkey@0"
I20260926 01:03:44.778649 24734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384372073861-3372-0/minicluster-data/master-0/wal/instance:
uuid: "e75d92b5931948fa86c0e8df498b908b"
format_stamp: "Formatted at 2026-09-26 01:03:44 on dist-test-slave-7v10"
server_key: "9794a8f3a0c31f265ec8ed5df856e106"
server_key_iv: "93dbc09d9841436782cff3826b0b9d74"
server_key_version: "encryptionkey@0"
I20260926 01:03:44.780026 24734 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:03:44.780745 24749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:44.780913 24734 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:03:44.781011 24734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384372073861-3372-0/minicluster-data/master-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384372073861-3372-0/minicluster-data/master-0/wal
uuid: "e75d92b5931948fa86c0e8df498b908b"
format_stamp: "Formatted at 2026-09-26 01:03:44 on dist-test-slave-7v10"
server_key: "9794a8f3a0c31f265ec8ed5df856e106"
server_key_iv: "93dbc09d9841436782cff3826b0b9d74"
server_key_version: "encryptionkey@0"
I20260926 01:03:44.781095 24734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384372073861-3372-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384372073861-3372-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384372073861-3372-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:03:44.796000 24734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:03:44.797756 24734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:03:44.797919 24734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:03:44.801829 24734 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43135
I20260926 01:03:44.801862 24801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43135 every 8 connection(s)
I20260926 01:03:44.802218 24734 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
I20260926 01:03:44.803119 24802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:03:44.805331 24802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e75d92b5931948fa86c0e8df498b908b: Bootstrap starting.
I20260926 01:03:44.805888 24802 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e75d92b5931948fa86c0e8df498b908b: Neither blocks nor log segments found. Creating new log.
I20260926 01:03:44.806161 24802 log.cc:824] T 00000000000000000000000000000000 P e75d92b5931948fa86c0e8df498b908b: Log is configured to *not* fsync() on all Append() calls
I20260926 01:03:44.806401 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 24734
I20260926 01:03:44.806537 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384372073861-3372-0/minicluster-data/master-0/wal/instance
I20260926 01:03:44.806754 3372 external_mini_cluster.cc:1428] Setting key bdbe82d98ae9350c74e2c777d27ccb2c
I20260926 01:03:44.807015 24802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e75d92b5931948fa86c0e8df498b908b: No bootstrap required, opened a new log
I20260926 01:03:44.808899 24802 raft_consensus.cc:348] T 00000000000000000000000000000000 P e75d92b5931948fa86c0e8df498b908b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e75d92b5931948fa86c0e8df498b908b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43135 } }
I20260926 01:03:44.809027 24802 raft_consensus.cc:374] T 00000000000000000000000000000000 P e75d92b5931948fa86c0e8df498b908b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:03:44.809051 24802 raft_consensus.cc:729] T 00000000000000000000000000000000 P e75d92b5931948fa86c0e8df498b908b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e75d92b5931948fa86c0e8df498b908b, State: Initialized, Role: FOLLOWER
I20260926 01:03:44.809191 24802 consensus_queue.cc:260] T 00000000000000000000000000000000 P e75d92b5931948fa86c0e8df498b908b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e75d92b5931948fa86c0e8df498b908b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43135 } }
I20260926 01:03:44.809264 24802 raft_consensus.cc:388] T 00000000000000000000000000000000 P e75d92b5931948fa86c0e8df498b908b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:03:44.809286 24802 raft_consensus.cc:482] T 00000000000000000000000000000000 P e75d92b5931948fa86c0e8df498b908b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:03:44.809351 24802 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e75d92b5931948fa86c0e8df498b908b [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:03:44.809890 24802 raft_consensus.cc:504] T 00000000000000000000000000000000 P e75d92b5931948fa86c0e8df498b908b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e75d92b5931948fa86c0e8df498b908b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43135 } }
I20260926 01:03:44.809970 24802 leader_election.cc:302] T 00000000000000000000000000000000 P e75d92b5931948fa86c0e8df498b908b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e75d92b5931948fa86c0e8df498b908b; no voters:
I20260926 01:03:44.810143 24802 leader_election.cc:288] T 00000000000000000000000000000000 P e75d92b5931948fa86c0e8df498b908b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:03:44.810175 24807 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e75d92b5931948fa86c0e8df498b908b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:03:44.810336 24807 raft_consensus.cc:686] T 00000000000000000000000000000000 P e75d92b5931948fa86c0e8df498b908b [term 1 LEADER]: Becoming Leader. State: Replica: e75d92b5931948fa86c0e8df498b908b, State: Running, Role: LEADER
I20260926 01:03:44.810392 24802 sys_catalog.cc:564] T 00000000000000000000000000000000 P e75d92b5931948fa86c0e8df498b908b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:03:44.810482 24807 consensus_queue.cc:237] T 00000000000000000000000000000000 P e75d92b5931948fa86c0e8df498b908b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e75d92b5931948fa86c0e8df498b908b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43135 } }
I20260926 01:03:44.810894 24809 sys_catalog.cc:455] T 00000000000000000000000000000000 P e75d92b5931948fa86c0e8df498b908b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e75d92b5931948fa86c0e8df498b908b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e75d92b5931948fa86c0e8df498b908b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43135 } } }
I20260926 01:03:44.810995 24809 sys_catalog.cc:458] T 00000000000000000000000000000000 P e75d92b5931948fa86c0e8df498b908b [sys.catalog]: This master's current role is: LEADER
I20260926 01:03:44.811816 24812 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:03:44.812006 24808 sys_catalog.cc:455] T 00000000000000000000000000000000 P e75d92b5931948fa86c0e8df498b908b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e75d92b5931948fa86c0e8df498b908b. Latest consensus state: current_term: 1 leader_uuid: "e75d92b5931948fa86c0e8df498b908b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e75d92b5931948fa86c0e8df498b908b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43135 } } }
I20260926 01:03:44.812111 24808 sys_catalog.cc:458] T 00000000000000000000000000000000 P e75d92b5931948fa86c0e8df498b908b [sys.catalog]: This master's current role is: LEADER
I20260926 01:03:44.812311 24812 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:03:44.813522 24806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:44.807309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.110:34879 (local address 127.0.0.1:43135)
0926 01:03:44.807498 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:44.807506 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:44.807536 (+ 30us) server_negotiation.cc:407] Connection header received
0926 01:03:44.808266 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:44.808275 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:44.808327 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:44.808477 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:44.808716 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:44.810070 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:44.811148 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:44.811825 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:44.811953 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:44.812861 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:44.812865 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:44.812865 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:03:44.813017 (+ 152us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:44.813047 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:03:44.813057 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:44.813165 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:44.813249 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 01:03:44.813277 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":84,"threads_started":1}
I20260926 01:03:44.813982 24812 catalog_manager.cc:1347] Generated new cluster ID: 7136f7e2e79f46a3a20085893668f499
I20260926 01:03:44.814034 24812 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:03:44.814414 3372 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:03:44.818624 24806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:44.815806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.110:37947 (local address 127.0.0.1:43135)
0926 01:03:44.815843 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:44.815845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:44.815956 (+ 111us) server_negotiation.cc:407] Connection header received
0926 01:03:44.816028 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:44.816029 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:44.816038 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:44.816093 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:44.816240 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:44.816696 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:44.817409 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:44.817639 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:44.817730 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:44.818386 (+ 656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:44.818387 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:44.818387 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:03:44.818392 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:44.818407 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:03:44.818411 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:44.818464 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:44.818523 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 01:03:44.818531 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 01:03:44.832047 24812 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:03:44.832738 24812 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:03:44.835811 24812 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e75d92b5931948fa86c0e8df498b908b: Generated new TSK 0
I20260926 01:03:44.835990 24812 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:03:44.839103 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 24734
W20260926 01:03:44.843387 24718 connection.cc:535] client connection to 127.0.0.1:43135 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43135): Connection reset by peer (error 104)
2026-09-26T01:03:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1301 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384372073861-3372-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:03:44 dist-test-slave-7v10 krb5kdc[24833](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:03:44 dist-test-slave-7v10 krb5kdc[24833](info): set up 2 sockets
Sep 26 01:03:44 dist-test-slave-7v10 krb5kdc[24833](info): commencing operation
krb5kdc: starting...
W20260926 01:03:45.878230 3372 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Sep 26 01:03:45 dist-test-slave-7v10 krb5kdc[24833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:03:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:03:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384372073861-3372-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384372073861-3372-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 01:03:45.993892 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384372073861-3372-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384372073861-3372-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384372073861-3372-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43129
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384372073861-3372-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43129
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:03:46.074061 24846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:03:46.074299 24846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:03:46.074338 24846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:03:46.075491 24846 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:03:46.075562 24846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:03:46.075587 24846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:03:46.075615 24846 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:03:46.075646 24846 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:03:46.077081 24846 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384372073861-3372-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384372073861-3372-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43129
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384372073861-3372-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384372073861-3372-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:03:46.077348 24846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:03:46.077581 24846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:03:46.082441 24854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:46.082435 24852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:46.082470 24851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:03:46.082743 24846 server_base.cc:1034] running on GCE node
I20260926 01:03:46.083003 24846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:03:46.083292 24846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:03:46.084455 24846 hybrid_clock.cc:648] HybridClock initialized: now 1790384626084443 us; error 29 us; skew 500 ppm
Sep 26 01:03:46 dist-test-slave-7v10 krb5kdc[24833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:03:46.086781 24846 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 01:03:46.087260 24846 webserver.cc:460] Webserver started at http://127.0.0.1:36679/ using document root <none> and password file <none>
I20260926 01:03:46.087497 24846 fs_manager.cc:359] Metadata directory not provided
I20260926 01:03:46.087566 24846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:03:46.087725 24846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:03:46.088749 24846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384372073861-3372-0/minicluster-data/master-0/data/instance:
uuid: "23f954118c3549da860ef70b3b917de2"
format_stamp: "Formatted at 2026-09-26 01:03:46 on dist-test-slave-7v10"
server_key: "259696804730d53a82330b04cf1e4270"
server_key_iv: "db400fbf847a5e4f03d43ea1a02fe2c8"
server_key_version: "encryptionkey@0"
I20260926 01:03:46.089100 24846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384372073861-3372-0/minicluster-data/master-0/wal/instance:
uuid: "23f954118c3549da860ef70b3b917de2"
format_stamp: "Formatted at 2026-09-26 01:03:46 on dist-test-slave-7v10"
server_key: "259696804730d53a82330b04cf1e4270"
server_key_iv: "db400fbf847a5e4f03d43ea1a02fe2c8"
server_key_version: "encryptionkey@0"
I20260926 01:03:46.090399 24846 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 01:03:46.091267 24861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:46.091428 24846 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:03:46.091531 24846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384372073861-3372-0/minicluster-data/master-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384372073861-3372-0/minicluster-data/master-0/wal
uuid: "23f954118c3549da860ef70b3b917de2"
format_stamp: "Formatted at 2026-09-26 01:03:46 on dist-test-slave-7v10"
server_key: "259696804730d53a82330b04cf1e4270"
server_key_iv: "db400fbf847a5e4f03d43ea1a02fe2c8"
server_key_version: "encryptionkey@0"
I20260926 01:03:46.091604 24846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384372073861-3372-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384372073861-3372-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384372073861-3372-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:03:46.094973 24846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:03:46.096771 24846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:03:46.096922 24846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:03:46.101110 24846 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43129
I20260926 01:03:46.101166 24914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43129 every 8 connection(s)
I20260926 01:03:46.101504 24846 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
I20260926 01:03:46.102716 24915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:03:46.104987 24915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23f954118c3549da860ef70b3b917de2: Bootstrap starting.
I20260926 01:03:46.105580 24915 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23f954118c3549da860ef70b3b917de2: Neither blocks nor log segments found. Creating new log.
I20260926 01:03:46.105870 24915 log.cc:824] T 00000000000000000000000000000000 P 23f954118c3549da860ef70b3b917de2: Log is configured to *not* fsync() on all Append() calls
I20260926 01:03:46.106599 24915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23f954118c3549da860ef70b3b917de2: No bootstrap required, opened a new log
I20260926 01:03:46.108219 24915 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23f954118c3549da860ef70b3b917de2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23f954118c3549da860ef70b3b917de2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43129 } }
I20260926 01:03:46.108353 24915 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23f954118c3549da860ef70b3b917de2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:03:46.108376 24915 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23f954118c3549da860ef70b3b917de2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23f954118c3549da860ef70b3b917de2, State: Initialized, Role: FOLLOWER
I20260926 01:03:46.108551 24915 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23f954118c3549da860ef70b3b917de2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23f954118c3549da860ef70b3b917de2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43129 } }
I20260926 01:03:46.108639 24915 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23f954118c3549da860ef70b3b917de2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:03:46.108695 24915 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23f954118c3549da860ef70b3b917de2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:03:46.108738 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 24846
I20260926 01:03:46.108765 24915 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23f954118c3549da860ef70b3b917de2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:03:46.108892 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384372073861-3372-0/minicluster-data/master-0/wal/instance
I20260926 01:03:46.109099 3372 external_mini_cluster.cc:1428] Setting key 0fbcbcaa6d1aff10a819212ee534685a
I20260926 01:03:46.109511 24915 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23f954118c3549da860ef70b3b917de2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23f954118c3549da860ef70b3b917de2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43129 } }
I20260926 01:03:46.109632 24915 leader_election.cc:302] T 00000000000000000000000000000000 P 23f954118c3549da860ef70b3b917de2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23f954118c3549da860ef70b3b917de2; no voters:
I20260926 01:03:46.109875 24915 leader_election.cc:288] T 00000000000000000000000000000000 P 23f954118c3549da860ef70b3b917de2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:03:46.109898 24920 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23f954118c3549da860ef70b3b917de2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:03:46.110008 24920 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23f954118c3549da860ef70b3b917de2 [term 1 LEADER]: Becoming Leader. State: Replica: 23f954118c3549da860ef70b3b917de2, State: Running, Role: LEADER
I20260926 01:03:46.110170 24915 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23f954118c3549da860ef70b3b917de2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:03:46.110134 24920 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23f954118c3549da860ef70b3b917de2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23f954118c3549da860ef70b3b917de2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43129 } }
I20260926 01:03:46.112177 24920 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23f954118c3549da860ef70b3b917de2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23f954118c3549da860ef70b3b917de2. Latest consensus state: current_term: 1 leader_uuid: "23f954118c3549da860ef70b3b917de2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23f954118c3549da860ef70b3b917de2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43129 } } }
I20260926 01:03:46.112407 24920 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23f954118c3549da860ef70b3b917de2 [sys.catalog]: This master's current role is: LEADER
I20260926 01:03:46.112807 24919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:46.109603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.110:44447 (local address 127.0.0.1:43129)
0926 01:03:46.109770 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:46.109778 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:46.109805 (+ 27us) server_negotiation.cc:407] Connection header received
0926 01:03:46.109944 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:46.109951 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:46.109983 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:46.110187 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:46.112253 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:46.112257 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:46.112258 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:03:46.112262 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:46.112307 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:03:46.112322 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:46.112381 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:46.112557 (+ 176us) server_negotiation.cc:299] Negotiation successful
0926 01:03:46.112605 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":79,"threads_started":1}
W20260926 01:03:46.112845 24934 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 23f954118c3549da860ef70b3b917de2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:03:46.112883 24934 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:03:46.113108 24922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23f954118c3549da860ef70b3b917de2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23f954118c3549da860ef70b3b917de2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23f954118c3549da860ef70b3b917de2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43129 } } }
I20260926 01:03:46.113212 24922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23f954118c3549da860ef70b3b917de2 [sys.catalog]: This master's current role is: LEADER
I20260926 01:03:46.113124 24936 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:03:46.113382 24936 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:03:46.113736 3372 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:03:46.115064 24936 catalog_manager.cc:1347] Generated new cluster ID: 914f729445554724aab04e28e246f45e
I20260926 01:03:46.115123 24936 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:03:46.116421 24919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:46.115346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.110:59237 (local address 127.0.0.1:43129)
0926 01:03:46.115384 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:46.115386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:46.115528 (+ 142us) server_negotiation.cc:407] Connection header received
0926 01:03:46.115570 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:46.115571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:46.115576 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:46.115620 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:46.116184 (+ 564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:46.116185 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:46.116185 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:03:46.116189 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:46.116202 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:03:46.116206 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:46.116269 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:46.116339 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 01:03:46.116352 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 01:03:46.122145 24936 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:03:46.122296 24936 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:03:46.130862 24936 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23f954118c3549da860ef70b3b917de2: Generated new TSK 0
I20260926 01:03:46.131033 24936 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:03:46.136010 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 24846
2026-09-26T01:03:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1296 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384372073861-3372-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:03:46 dist-test-slave-7v10 krb5kdc[24946](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:03:46 dist-test-slave-7v10 krb5kdc[24946](info): set up 2 sockets
Sep 26 01:03:46 dist-test-slave-7v10 krb5kdc[24946](info): commencing operation
krb5kdc: starting...
W20260926 01:03:47.173909 3372 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Sep 26 01:03:47 dist-test-slave-7v10 krb5kdc[24946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:03:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:03:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.75.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab.
Entry for principal kudu/127.3.75.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab.
I20260926 01:03:47.290023 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384372073861-3372-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384372073861-3372-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384372073861-3372-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:38729
--webserver_interface=127.3.75.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:45143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.75.62:38729
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:03:47.372335 24959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:03:47.372579 24959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:03:47.372613 24959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:03:47.373847 24959 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:03:47.373907 24959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:03:47.373931 24959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:03:47.373963 24959 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:03:47.373987 24959 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:03:47.375522 24959 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:45143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384372073861-3372-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384372073861-3372-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.75.62:38729
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:38729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384372073861-3372-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:03:47.375797 24959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:03:47.376014 24959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:03:47.380774 24967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:47.380776 24964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:47.380918 24965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:03:47.381039 24959 server_base.cc:1034] running on GCE node
I20260926 01:03:47.381260 24959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:03:47.381549 24959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:03:47.382714 24959 hybrid_clock.cc:648] HybridClock initialized: now 1790384627382700 us; error 45 us; skew 500 ppm
Sep 26 01:03:47 dist-test-slave-7v10 krb5kdc[24946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384627, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:03:47.385102 24959 init.cc:377] Logged in from keytab as kudu/127.3.75.62@KRBTEST.COM (short username kudu)
I20260926 01:03:47.385555 24959 webserver.cc:460] Webserver started at http://127.3.75.62:45687/ using document root <none> and password file <none>
I20260926 01:03:47.385782 24959 fs_manager.cc:359] Metadata directory not provided
I20260926 01:03:47.385849 24959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:03:47.386003 24959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:03:47.387121 24959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384372073861-3372-0/minicluster-data/master-0/data/instance:
uuid: "73f8ad4325ee48aaa09796be26dc6b5b"
format_stamp: "Formatted at 2026-09-26 01:03:47 on dist-test-slave-7v10"
server_key: "27431467845d7bfd28137600570c3bcf"
server_key_iv: "fc7e2a3e5c6ece221b4f5ea9618ee726"
server_key_version: "encryptionkey@0"
I20260926 01:03:47.387470 24959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384372073861-3372-0/minicluster-data/master-0/wal/instance:
uuid: "73f8ad4325ee48aaa09796be26dc6b5b"
format_stamp: "Formatted at 2026-09-26 01:03:47 on dist-test-slave-7v10"
server_key: "27431467845d7bfd28137600570c3bcf"
server_key_iv: "fc7e2a3e5c6ece221b4f5ea9618ee726"
server_key_version: "encryptionkey@0"
I20260926 01:03:47.388792 24959 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:03:47.389516 24974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:47.389681 24959 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:03:47.389777 24959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384372073861-3372-0/minicluster-data/master-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384372073861-3372-0/minicluster-data/master-0/wal
uuid: "73f8ad4325ee48aaa09796be26dc6b5b"
format_stamp: "Formatted at 2026-09-26 01:03:47 on dist-test-slave-7v10"
server_key: "27431467845d7bfd28137600570c3bcf"
server_key_iv: "fc7e2a3e5c6ece221b4f5ea9618ee726"
server_key_version: "encryptionkey@0"
I20260926 01:03:47.389858 24959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384372073861-3372-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384372073861-3372-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384372073861-3372-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:03:47.402786 24959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:03:47.404609 24959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:03:47.404773 24959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:03:47.408658 24959 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.62:38729
I20260926 01:03:47.408685 25027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.62:38729 every 8 connection(s)
I20260926 01:03:47.409019 24959 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
I20260926 01:03:47.409811 25028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:03:47.412042 25028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73f8ad4325ee48aaa09796be26dc6b5b: Bootstrap starting.
I20260926 01:03:47.412623 25028 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73f8ad4325ee48aaa09796be26dc6b5b: Neither blocks nor log segments found. Creating new log.
I20260926 01:03:47.412902 25028 log.cc:824] T 00000000000000000000000000000000 P 73f8ad4325ee48aaa09796be26dc6b5b: Log is configured to *not* fsync() on all Append() calls
I20260926 01:03:47.413547 25028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73f8ad4325ee48aaa09796be26dc6b5b: No bootstrap required, opened a new log
I20260926 01:03:47.415093 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 24959
I20260926 01:03:47.415220 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384372073861-3372-0/minicluster-data/master-0/wal/instance
I20260926 01:03:47.415282 25028 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73f8ad4325ee48aaa09796be26dc6b5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73f8ad4325ee48aaa09796be26dc6b5b" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 38729 } }
I20260926 01:03:47.415417 3372 external_mini_cluster.cc:1428] Setting key 0d693e4dae7751d702395c2a7d2611e5
I20260926 01:03:47.415453 25028 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73f8ad4325ee48aaa09796be26dc6b5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:03:47.415491 25028 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73f8ad4325ee48aaa09796be26dc6b5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73f8ad4325ee48aaa09796be26dc6b5b, State: Initialized, Role: FOLLOWER
I20260926 01:03:47.415634 25028 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73f8ad4325ee48aaa09796be26dc6b5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73f8ad4325ee48aaa09796be26dc6b5b" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 38729 } }
I20260926 01:03:47.415724 25028 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73f8ad4325ee48aaa09796be26dc6b5b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:03:47.415761 25028 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73f8ad4325ee48aaa09796be26dc6b5b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:03:47.415866 25028 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73f8ad4325ee48aaa09796be26dc6b5b [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:03:47.416662 25028 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73f8ad4325ee48aaa09796be26dc6b5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73f8ad4325ee48aaa09796be26dc6b5b" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 38729 } }
I20260926 01:03:47.416771 25028 leader_election.cc:302] T 00000000000000000000000000000000 P 73f8ad4325ee48aaa09796be26dc6b5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73f8ad4325ee48aaa09796be26dc6b5b; no voters:
I20260926 01:03:47.416944 25028 leader_election.cc:288] T 00000000000000000000000000000000 P 73f8ad4325ee48aaa09796be26dc6b5b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:03:47.416965 25033 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73f8ad4325ee48aaa09796be26dc6b5b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:03:47.417073 25033 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73f8ad4325ee48aaa09796be26dc6b5b [term 1 LEADER]: Becoming Leader. State: Replica: 73f8ad4325ee48aaa09796be26dc6b5b, State: Running, Role: LEADER
I20260926 01:03:47.417207 25028 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73f8ad4325ee48aaa09796be26dc6b5b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:03:47.417204 25033 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73f8ad4325ee48aaa09796be26dc6b5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73f8ad4325ee48aaa09796be26dc6b5b" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 38729 } }
I20260926 01:03:47.417671 25034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73f8ad4325ee48aaa09796be26dc6b5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "73f8ad4325ee48aaa09796be26dc6b5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73f8ad4325ee48aaa09796be26dc6b5b" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 38729 } } }
I20260926 01:03:47.417953 25034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73f8ad4325ee48aaa09796be26dc6b5b [sys.catalog]: This master's current role is: LEADER
I20260926 01:03:47.417868 25035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73f8ad4325ee48aaa09796be26dc6b5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73f8ad4325ee48aaa09796be26dc6b5b. Latest consensus state: current_term: 1 leader_uuid: "73f8ad4325ee48aaa09796be26dc6b5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73f8ad4325ee48aaa09796be26dc6b5b" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 38729 } } }
I20260926 01:03:47.418344 25035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73f8ad4325ee48aaa09796be26dc6b5b [sys.catalog]: This master's current role is: LEADER
I20260926 01:03:47.418464 25042 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:03:47.419036 25042 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:03:47.420615 25042 catalog_manager.cc:1347] Generated new cluster ID: 047cd38a19974271a04b81236e13dd67
I20260926 01:03:47.420670 25042 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 01:03:47 dist-test-slave-7v10 krb5kdc[24946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 01:03:47.428161 25032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:47.415867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57405 (local address 127.3.75.62:38729)
0926 01:03:47.416074 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:47.416084 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:47.416118 (+ 34us) server_negotiation.cc:407] Connection header received
0926 01:03:47.416527 (+ 409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:47.416536 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:47.416596 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:47.416762 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:47.417568 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:47.419081 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:47.420067 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:47.420639 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:47.420745 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:47.423714 (+ 2969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:47.423717 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:47.423718 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:03:47.423729 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:47.426258 (+ 2529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:47.426705 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:47.426708 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:47.426708 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:47.426822 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:47.427143 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:47.427144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:47.427144 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:47.427459 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:47.427609 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:47.427795 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 01:03:47.427902 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":156,"thread_start_us":105,"threads_started":1}
I20260926 01:03:47.428196 25042 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:03:47.428978 25042 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:03:47.429307 3372 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:03:47.435384 25042 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73f8ad4325ee48aaa09796be26dc6b5b: Generated new TSK 0
I20260926 01:03:47.435535 25042 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:03:47.440685 25032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:47.431123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41295 (local address 127.3.75.62:38729)
0926 01:03:47.431157 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:47.431159 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:47.434032 (+ 2873us) server_negotiation.cc:407] Connection header received
0926 01:03:47.434105 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:47.434106 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:47.434115 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:47.434168 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:47.434799 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:47.435594 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:47.436608 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:47.436824 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:47.436877 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:47.438225 (+ 1348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:47.438227 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:47.438227 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:03:47.438232 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:47.439615 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:47.440007 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:47.440008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:47.440008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:47.440057 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:47.440277 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:47.440277 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:47.440277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:47.440366 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:47.440436 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:47.440586 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 01:03:47.440631 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 01:03:47.442504 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 24959
W20260926 01:03:47.446863 24944 connection.cc:535] client connection to 127.3.75.62:38729 recv error: Network error: failed to read from TLS socket (remote: 127.3.75.62:38729): Connection reset by peer (error 104)
2026-09-26T01:03:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1306 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384372073861-3372-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:03:47 dist-test-slave-7v10 krb5kdc[25059](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:03:47 dist-test-slave-7v10 krb5kdc[25059](info): set up 2 sockets
Sep 26 01:03:47 dist-test-slave-7v10 krb5kdc[25059](info): commencing operation
krb5kdc: starting...
W20260926 01:03:48.478133 3372 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Sep 26 01:03:48 dist-test-slave-7v10 krb5kdc[25059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:03:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:03:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.75.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab.
Entry for principal kudu/127.3.75.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab.
I20260926 01:03:48.591970 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384372073861-3372-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384372073861-3372-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384372073861-3372-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:35511
--webserver_interface=127.3.75.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:38739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.75.62:35511
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:03:48.672027 25072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:03:48.672223 25072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:03:48.672245 25072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:03:48.673373 25072 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:03:48.673416 25072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:03:48.673430 25072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:03:48.673470 25072 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:03:48.673501 25072 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:03:48.675019 25072 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:38739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384372073861-3372-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384372073861-3372-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.75.62:35511
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:35511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384372073861-3372-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:03:48.675267 25072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:03:48.675486 25072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:03:48.680472 25077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:48.680485 25080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:03:48.680644 25072 server_base.cc:1034] running on GCE node
W20260926 01:03:48.680485 25078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:03:48.680980 25072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:03:48.681224 25072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:03:48.682374 25072 hybrid_clock.cc:648] HybridClock initialized: now 1790384628682363 us; error 28 us; skew 500 ppm
Sep 26 01:03:48 dist-test-slave-7v10 krb5kdc[25059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384628, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:03:48.688256 25072 init.cc:377] Logged in from keytab as kudu/127.3.75.62@KRBTEST.COM (short username kudu)
I20260926 01:03:48.688695 25072 webserver.cc:460] Webserver started at http://127.3.75.62:36911/ using document root <none> and password file <none>
I20260926 01:03:48.688872 25072 fs_manager.cc:359] Metadata directory not provided
I20260926 01:03:48.688910 25072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:03:48.688997 25072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:03:48.689957 25072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384372073861-3372-0/minicluster-data/master-0/data/instance:
uuid: "efd217b88a514eab95b285dcc4b58360"
format_stamp: "Formatted at 2026-09-26 01:03:48 on dist-test-slave-7v10"
server_key: "70eade0b7bd77e2a5941f34f90528b43"
server_key_iv: "517c2a4abeb9a90b5b93563d3944ea5d"
server_key_version: "encryptionkey@0"
I20260926 01:03:48.690241 25072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384372073861-3372-0/minicluster-data/master-0/wal/instance:
uuid: "efd217b88a514eab95b285dcc4b58360"
format_stamp: "Formatted at 2026-09-26 01:03:48 on dist-test-slave-7v10"
server_key: "70eade0b7bd77e2a5941f34f90528b43"
server_key_iv: "517c2a4abeb9a90b5b93563d3944ea5d"
server_key_version: "encryptionkey@0"
I20260926 01:03:48.691542 25072 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:03:48.692229 25087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:48.692402 25072 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:03:48.692510 25072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384372073861-3372-0/minicluster-data/master-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384372073861-3372-0/minicluster-data/master-0/wal
uuid: "efd217b88a514eab95b285dcc4b58360"
format_stamp: "Formatted at 2026-09-26 01:03:48 on dist-test-slave-7v10"
server_key: "70eade0b7bd77e2a5941f34f90528b43"
server_key_iv: "517c2a4abeb9a90b5b93563d3944ea5d"
server_key_version: "encryptionkey@0"
I20260926 01:03:48.692592 25072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384372073861-3372-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384372073861-3372-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384372073861-3372-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:03:48.705564 25072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:03:48.707480 25072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:03:48.707638 25072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:03:48.711598 25072 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.62:35511
I20260926 01:03:48.711627 25139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.62:35511 every 8 connection(s)
I20260926 01:03:48.711959 25072 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
I20260926 01:03:48.712754 25140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:03:48.714865 25140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efd217b88a514eab95b285dcc4b58360: Bootstrap starting.
I20260926 01:03:48.715443 25140 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P efd217b88a514eab95b285dcc4b58360: Neither blocks nor log segments found. Creating new log.
I20260926 01:03:48.715684 25140 log.cc:824] T 00000000000000000000000000000000 P efd217b88a514eab95b285dcc4b58360: Log is configured to *not* fsync() on all Append() calls
I20260926 01:03:48.716282 25140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efd217b88a514eab95b285dcc4b58360: No bootstrap required, opened a new log
I20260926 01:03:48.716286 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 25072
I20260926 01:03:48.716416 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384372073861-3372-0/minicluster-data/master-0/wal/instance
I20260926 01:03:48.716610 3372 external_mini_cluster.cc:1428] Setting key 5ac0f42151fd5400736bd965ba78a169
I20260926 01:03:48.718852 25140 raft_consensus.cc:348] T 00000000000000000000000000000000 P efd217b88a514eab95b285dcc4b58360 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efd217b88a514eab95b285dcc4b58360" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 35511 } }
I20260926 01:03:48.719058 25140 raft_consensus.cc:374] T 00000000000000000000000000000000 P efd217b88a514eab95b285dcc4b58360 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:03:48.719100 25140 raft_consensus.cc:729] T 00000000000000000000000000000000 P efd217b88a514eab95b285dcc4b58360 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efd217b88a514eab95b285dcc4b58360, State: Initialized, Role: FOLLOWER
I20260926 01:03:48.719251 25140 consensus_queue.cc:260] T 00000000000000000000000000000000 P efd217b88a514eab95b285dcc4b58360 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efd217b88a514eab95b285dcc4b58360" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 35511 } }
I20260926 01:03:48.719353 25140 raft_consensus.cc:388] T 00000000000000000000000000000000 P efd217b88a514eab95b285dcc4b58360 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:03:48.719416 25140 raft_consensus.cc:482] T 00000000000000000000000000000000 P efd217b88a514eab95b285dcc4b58360 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:03:48.719465 25140 raft_consensus.cc:3037] T 00000000000000000000000000000000 P efd217b88a514eab95b285dcc4b58360 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:03:48.720122 25140 raft_consensus.cc:504] T 00000000000000000000000000000000 P efd217b88a514eab95b285dcc4b58360 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efd217b88a514eab95b285dcc4b58360" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 35511 } }
I20260926 01:03:48.720228 25140 leader_election.cc:302] T 00000000000000000000000000000000 P efd217b88a514eab95b285dcc4b58360 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: efd217b88a514eab95b285dcc4b58360; no voters:
I20260926 01:03:48.720372 25140 leader_election.cc:288] T 00000000000000000000000000000000 P efd217b88a514eab95b285dcc4b58360 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:03:48.720456 25145 raft_consensus.cc:2781] T 00000000000000000000000000000000 P efd217b88a514eab95b285dcc4b58360 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:03:48.720585 25140 sys_catalog.cc:564] T 00000000000000000000000000000000 P efd217b88a514eab95b285dcc4b58360 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:03:48.720633 25145 raft_consensus.cc:686] T 00000000000000000000000000000000 P efd217b88a514eab95b285dcc4b58360 [term 1 LEADER]: Becoming Leader. State: Replica: efd217b88a514eab95b285dcc4b58360, State: Running, Role: LEADER
I20260926 01:03:48.720722 25145 consensus_queue.cc:237] T 00000000000000000000000000000000 P efd217b88a514eab95b285dcc4b58360 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efd217b88a514eab95b285dcc4b58360" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 35511 } }
I20260926 01:03:48.721798 25147 sys_catalog.cc:455] T 00000000000000000000000000000000 P efd217b88a514eab95b285dcc4b58360 [sys.catalog]: SysCatalogTable state changed. Reason: New leader efd217b88a514eab95b285dcc4b58360. Latest consensus state: current_term: 1 leader_uuid: "efd217b88a514eab95b285dcc4b58360" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efd217b88a514eab95b285dcc4b58360" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 35511 } } }
I20260926 01:03:48.721892 25147 sys_catalog.cc:458] T 00000000000000000000000000000000 P efd217b88a514eab95b285dcc4b58360 [sys.catalog]: This master's current role is: LEADER
W20260926 01:03:48.722357 25161 catalog_manager.cc:1558] T 00000000000000000000000000000000 P efd217b88a514eab95b285dcc4b58360: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260926 01:03:48.722360 25146 sys_catalog.cc:455] T 00000000000000000000000000000000 P efd217b88a514eab95b285dcc4b58360 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "efd217b88a514eab95b285dcc4b58360" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efd217b88a514eab95b285dcc4b58360" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 35511 } } }
I20260926 01:03:48.722535 25146 sys_catalog.cc:458] T 00000000000000000000000000000000 P efd217b88a514eab95b285dcc4b58360 [sys.catalog]: This master's current role is: LEADER
W20260926 01:03:48.722477 25161 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:03:48.722672 25162 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:03:48.722934 25162 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:03:48.724298 25144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:48.717082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43383 (local address 127.3.75.62:35511)
0926 01:03:48.717279 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:48.717288 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:48.717320 (+ 32us) server_negotiation.cc:407] Connection header received
0926 01:03:48.718806 (+ 1486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:48.718815 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:48.718877 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:48.719050 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:48.719395 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:48.720757 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:48.721968 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:48.722565 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:48.722671 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:48.723624 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:48.723629 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:48.723629 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:03:48.723830 (+ 201us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:48.723868 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:03:48.723882 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:48.723969 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:48.724072 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 01:03:48.724101 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":87,"threads_started":1}
I20260926 01:03:48.724785 25162 catalog_manager.cc:1347] Generated new cluster ID: d4504740ed3645b892e90bc1d3b8bde6
I20260926 01:03:48.724834 25162 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:03:48.725178 3372 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:03:48.729328 25144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:48.726479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59409 (local address 127.3.75.62:35511)
0926 01:03:48.726518 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:48.726520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:48.726641 (+ 121us) server_negotiation.cc:407] Connection header received
0926 01:03:48.726713 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:48.726714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:48.726724 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:48.726768 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:48.726945 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:48.727400 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:48.728109 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:48.728330 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:48.728418 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:48.729084 (+ 666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:48.729085 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:48.729085 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:03:48.729089 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:48.729103 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:03:48.729107 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:48.729160 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:48.729225 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 01:03:48.729234 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 01:03:48.737437 25162 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:03:48.737987 25162 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:03:48.749800 25162 catalog_manager.cc:5868] T 00000000000000000000000000000000 P efd217b88a514eab95b285dcc4b58360: Generated new TSK 0
I20260926 01:03:48.749995 25162 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:03:48.757232 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 25072
W20260926 01:03:48.761482 25057 connection.cc:535] client connection to 127.3.75.62:35511 recv error: Network error: failed to read from TLS socket (remote: 127.3.75.62:35511): Connection reset by peer (error 104)
2026-09-26T01:03:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1314 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384372073861-3372-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:03:48 dist-test-slave-7v10 krb5kdc[25172](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:03:48 dist-test-slave-7v10 krb5kdc[25172](info): set up 2 sockets
Sep 26 01:03:48 dist-test-slave-7v10 krb5kdc[25172](info): commencing operation
krb5kdc: starting...
W20260926 01:03:49.789937 3372 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Sep 26 01:03:49 dist-test-slave-7v10 krb5kdc[25172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:03:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:03:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.75.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab.
Entry for principal kudu/127.3.75.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab.
I20260926 01:03:49.905407 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384372073861-3372-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384372073861-3372-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384372073861-3372-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:43381
--webserver_interface=127.3.75.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:42615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.75.62:43381
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:03:49.985704 25185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:03:49.985908 25185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:03:49.985932 25185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:03:49.987069 25185 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:03:49.987105 25185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:03:49.987119 25185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:03:49.987133 25185 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:03:49.987145 25185 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:03:49.988570 25185 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:42615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384372073861-3372-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384372073861-3372-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.75.62:43381
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:43381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384372073861-3372-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:03:49.988759 25185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:03:49.988930 25185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:03:49.993974 25191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:49.994007 25190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:49.994057 25193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:03:49.994165 25185 server_base.cc:1034] running on GCE node
I20260926 01:03:49.994395 25185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:03:49.994747 25185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:03:49.995931 25185 hybrid_clock.cc:648] HybridClock initialized: now 1790384629995922 us; error 39 us; skew 500 ppm
Sep 26 01:03:49 dist-test-slave-7v10 krb5kdc[25172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384629, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:03:49.998379 25185 init.cc:377] Logged in from keytab as kudu/127.3.75.62@KRBTEST.COM (short username kudu)
I20260926 01:03:49.998835 25185 webserver.cc:460] Webserver started at http://127.3.75.62:35961/ using document root <none> and password file <none>
I20260926 01:03:49.999101 25185 fs_manager.cc:359] Metadata directory not provided
I20260926 01:03:49.999172 25185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:03:49.999333 25185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:03:50.000455 25185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384372073861-3372-0/minicluster-data/master-0/data/instance:
uuid: "cc9dada3f92b4a648f2f40b7b3a1107a"
format_stamp: "Formatted at 2026-09-26 01:03:49 on dist-test-slave-7v10"
server_key: "4ed7a92bb4ee97c3edb511e49c94694a"
server_key_iv: "1b12a9e80ffe8519a7ca8ee171fed067"
server_key_version: "encryptionkey@0"
I20260926 01:03:50.000829 25185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384372073861-3372-0/minicluster-data/master-0/wal/instance:
uuid: "cc9dada3f92b4a648f2f40b7b3a1107a"
format_stamp: "Formatted at 2026-09-26 01:03:49 on dist-test-slave-7v10"
server_key: "4ed7a92bb4ee97c3edb511e49c94694a"
server_key_iv: "1b12a9e80ffe8519a7ca8ee171fed067"
server_key_version: "encryptionkey@0"
I20260926 01:03:50.002173 25185 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:03:50.002933 25200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:50.003106 25185 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:03:50.003206 25185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384372073861-3372-0/minicluster-data/master-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384372073861-3372-0/minicluster-data/master-0/wal
uuid: "cc9dada3f92b4a648f2f40b7b3a1107a"
format_stamp: "Formatted at 2026-09-26 01:03:49 on dist-test-slave-7v10"
server_key: "4ed7a92bb4ee97c3edb511e49c94694a"
server_key_iv: "1b12a9e80ffe8519a7ca8ee171fed067"
server_key_version: "encryptionkey@0"
I20260926 01:03:50.003294 25185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384372073861-3372-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384372073861-3372-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384372073861-3372-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:03:50.006482 25185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:03:50.008455 25185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:03:50.008605 25185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:03:50.012416 25185 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.62:43381
I20260926 01:03:50.012465 25252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.62:43381 every 8 connection(s)
I20260926 01:03:50.012780 25185 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
I20260926 01:03:50.013760 25253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:03:50.016011 25253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc9dada3f92b4a648f2f40b7b3a1107a: Bootstrap starting.
I20260926 01:03:50.016574 25253 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc9dada3f92b4a648f2f40b7b3a1107a: Neither blocks nor log segments found. Creating new log.
I20260926 01:03:50.016853 25253 log.cc:824] T 00000000000000000000000000000000 P cc9dada3f92b4a648f2f40b7b3a1107a: Log is configured to *not* fsync() on all Append() calls
I20260926 01:03:50.017532 25253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc9dada3f92b4a648f2f40b7b3a1107a: No bootstrap required, opened a new log
I20260926 01:03:50.019129 25253 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc9dada3f92b4a648f2f40b7b3a1107a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc9dada3f92b4a648f2f40b7b3a1107a" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 43381 } }
I20260926 01:03:50.019309 25253 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc9dada3f92b4a648f2f40b7b3a1107a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:03:50.019373 25253 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc9dada3f92b4a648f2f40b7b3a1107a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc9dada3f92b4a648f2f40b7b3a1107a, State: Initialized, Role: FOLLOWER
I20260926 01:03:50.019559 25253 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc9dada3f92b4a648f2f40b7b3a1107a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc9dada3f92b4a648f2f40b7b3a1107a" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 43381 } }
I20260926 01:03:50.019660 25253 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc9dada3f92b4a648f2f40b7b3a1107a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:03:50.019704 25253 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc9dada3f92b4a648f2f40b7b3a1107a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:03:50.019754 25253 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc9dada3f92b4a648f2f40b7b3a1107a [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:03:50.019994 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 25185
I20260926 01:03:50.020125 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384372073861-3372-0/minicluster-data/master-0/wal/instance
I20260926 01:03:50.020332 3372 external_mini_cluster.cc:1428] Setting key 64fd83019ec4bde9c79f3bceb6be4360
I20260926 01:03:50.020893 25253 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc9dada3f92b4a648f2f40b7b3a1107a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc9dada3f92b4a648f2f40b7b3a1107a" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 43381 } }
I20260926 01:03:50.021091 25253 leader_election.cc:302] T 00000000000000000000000000000000 P cc9dada3f92b4a648f2f40b7b3a1107a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc9dada3f92b4a648f2f40b7b3a1107a; no voters:
I20260926 01:03:50.021281 25253 leader_election.cc:288] T 00000000000000000000000000000000 P cc9dada3f92b4a648f2f40b7b3a1107a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:03:50.021322 25258 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc9dada3f92b4a648f2f40b7b3a1107a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:03:50.021494 25258 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc9dada3f92b4a648f2f40b7b3a1107a [term 1 LEADER]: Becoming Leader. State: Replica: cc9dada3f92b4a648f2f40b7b3a1107a, State: Running, Role: LEADER
I20260926 01:03:50.021623 25258 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc9dada3f92b4a648f2f40b7b3a1107a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc9dada3f92b4a648f2f40b7b3a1107a" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 43381 } }
I20260926 01:03:50.021938 25253 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc9dada3f92b4a648f2f40b7b3a1107a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:03:50.021938 25258 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc9dada3f92b4a648f2f40b7b3a1107a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc9dada3f92b4a648f2f40b7b3a1107a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc9dada3f92b4a648f2f40b7b3a1107a" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 43381 } } }
I20260926 01:03:50.022058 25260 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc9dada3f92b4a648f2f40b7b3a1107a [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc9dada3f92b4a648f2f40b7b3a1107a. Latest consensus state: current_term: 1 leader_uuid: "cc9dada3f92b4a648f2f40b7b3a1107a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc9dada3f92b4a648f2f40b7b3a1107a" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 43381 } } }
I20260926 01:03:50.022189 25260 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc9dada3f92b4a648f2f40b7b3a1107a [sys.catalog]: This master's current role is: LEADER
I20260926 01:03:50.022120 25258 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc9dada3f92b4a648f2f40b7b3a1107a [sys.catalog]: This master's current role is: LEADER
I20260926 01:03:50.022668 25257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:50.020852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33597 (local address 127.3.75.62:43381)
0926 01:03:50.021031 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:50.021048 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:50.021081 (+ 33us) server_negotiation.cc:407] Connection header received
0926 01:03:50.021170 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:50.021175 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:50.021219 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:50.021377 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:50.022056 (+ 679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:50.022060 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:50.022060 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:03:50.022065 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:50.022107 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:03:50.022122 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:50.022185 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:50.022400 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 01:03:50.022441 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":99,"threads_started":1}
I20260926 01:03:50.023005 25262 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:03:50.023507 25262 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:03:50.023579 3372 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:03:50.025050 25262 catalog_manager.cc:1347] Generated new cluster ID: fb296bed66374b0889a7098c31bf0e92
I20260926 01:03:50.025104 25262 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:03:50.028076 25257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:50.025215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50627 (local address 127.3.75.62:43381)
0926 01:03:50.025251 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:50.025252 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:50.027103 (+ 1851us) server_negotiation.cc:407] Connection header received
0926 01:03:50.027183 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:50.027183 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:50.027188 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:50.027245 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:50.027867 (+ 622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:50.027867 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:50.027868 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:03:50.027872 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:50.027884 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:03:50.027888 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:50.027933 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:50.027985 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 01:03:50.027999 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 01:03:50.040333 25262 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:03:50.040530 25262 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:03:50.050518 25262 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc9dada3f92b4a648f2f40b7b3a1107a: Generated new TSK 0
I20260926 01:03:50.050743 25262 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:03:50.057222 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 25185
2026-09-26T01:03:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1300 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384372073861-3372-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:03:50 dist-test-slave-7v10 krb5kdc[25284](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:03:50 dist-test-slave-7v10 krb5kdc[25284](info): set up 2 sockets
Sep 26 01:03:50 dist-test-slave-7v10 krb5kdc[25284](info): commencing operation
krb5kdc: starting...
W20260926 01:03:51.086194 3372 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Sep 26 01:03:51 dist-test-slave-7v10 krb5kdc[25284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:03:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:03:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.75.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab.
Entry for principal kudu/127.3.75.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab.
I20260926 01:03:51.199908 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384372073861-3372-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384372073861-3372-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384372073861-3372-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:46625
--webserver_interface=127.3.75.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:37629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.75.62:46625
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:03:51.280128 25297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:03:51.280385 25297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:03:51.280424 25297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:03:51.281584 25297 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:03:51.281679 25297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:03:51.281711 25297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:03:51.281747 25297 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:03:51.281774 25297 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:03:51.283345 25297 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:37629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384372073861-3372-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384372073861-3372-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.75.62:46625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:46625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384372073861-3372-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:03:51.283641 25297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:03:51.283897 25297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:03:51.288822 25305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:51.288760 25303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:51.288776 25302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:03:51.289070 25297 server_base.cc:1034] running on GCE node
I20260926 01:03:51.289319 25297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:03:51.289613 25297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:03:51.290776 25297 hybrid_clock.cc:648] HybridClock initialized: now 1790384631290757 us; error 41 us; skew 500 ppm
Sep 26 01:03:51 dist-test-slave-7v10 krb5kdc[25284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384631, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:03:51.293237 25297 init.cc:377] Logged in from keytab as kudu/127.3.75.62@KRBTEST.COM (short username kudu)
I20260926 01:03:51.293668 25297 webserver.cc:460] Webserver started at http://127.3.75.62:38821/ using document root <none> and password file <none>
I20260926 01:03:51.293890 25297 fs_manager.cc:359] Metadata directory not provided
I20260926 01:03:51.293934 25297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:03:51.294019 25297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:03:51.295078 25297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384372073861-3372-0/minicluster-data/master-0/data/instance:
uuid: "4098661e3e364e75945c10b462433269"
format_stamp: "Formatted at 2026-09-26 01:03:51 on dist-test-slave-7v10"
server_key: "574d4619f8c171e77839d2f5c6c569c9"
server_key_iv: "7f0e6479d803a7a2c560c1463eb908c1"
server_key_version: "encryptionkey@0"
I20260926 01:03:51.295375 25297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384372073861-3372-0/minicluster-data/master-0/wal/instance:
uuid: "4098661e3e364e75945c10b462433269"
format_stamp: "Formatted at 2026-09-26 01:03:51 on dist-test-slave-7v10"
server_key: "574d4619f8c171e77839d2f5c6c569c9"
server_key_iv: "7f0e6479d803a7a2c560c1463eb908c1"
server_key_version: "encryptionkey@0"
I20260926 01:03:51.296579 25297 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:03:51.297226 25312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:51.297453 25297 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:03:51.297552 25297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384372073861-3372-0/minicluster-data/master-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384372073861-3372-0/minicluster-data/master-0/wal
uuid: "4098661e3e364e75945c10b462433269"
format_stamp: "Formatted at 2026-09-26 01:03:51 on dist-test-slave-7v10"
server_key: "574d4619f8c171e77839d2f5c6c569c9"
server_key_iv: "7f0e6479d803a7a2c560c1463eb908c1"
server_key_version: "encryptionkey@0"
I20260926 01:03:51.297623 25297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384372073861-3372-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384372073861-3372-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384372073861-3372-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:03:51.315234 25297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:03:51.317020 25297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:03:51.317174 25297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:03:51.320888 25297 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.62:46625
I20260926 01:03:51.320966 25366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.62:46625 every 8 connection(s)
I20260926 01:03:51.321276 25297 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
I20260926 01:03:51.322117 25367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:03:51.324286 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 25297
I20260926 01:03:51.324337 25367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4098661e3e364e75945c10b462433269: Bootstrap starting.
I20260926 01:03:51.324430 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384372073861-3372-0/minicluster-data/master-0/wal/instance
I20260926 01:03:51.324600 3372 external_mini_cluster.cc:1428] Setting key 7d676c33d2eb5bcd5213f8dfecef43e3
I20260926 01:03:51.325032 25367 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4098661e3e364e75945c10b462433269: Neither blocks nor log segments found. Creating new log.
I20260926 01:03:51.325390 25367 log.cc:824] T 00000000000000000000000000000000 P 4098661e3e364e75945c10b462433269: Log is configured to *not* fsync() on all Append() calls
I20260926 01:03:51.326112 25367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4098661e3e364e75945c10b462433269: No bootstrap required, opened a new log
I20260926 01:03:51.328292 25367 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4098661e3e364e75945c10b462433269 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4098661e3e364e75945c10b462433269" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 46625 } }
I20260926 01:03:51.328454 25367 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4098661e3e364e75945c10b462433269 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:03:51.328483 25367 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4098661e3e364e75945c10b462433269 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4098661e3e364e75945c10b462433269, State: Initialized, Role: FOLLOWER
I20260926 01:03:51.328593 25367 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4098661e3e364e75945c10b462433269 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4098661e3e364e75945c10b462433269" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 46625 } }
I20260926 01:03:51.328662 25367 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4098661e3e364e75945c10b462433269 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:03:51.328708 25367 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4098661e3e364e75945c10b462433269 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:03:51.328763 25367 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4098661e3e364e75945c10b462433269 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:03:51.329357 25367 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4098661e3e364e75945c10b462433269 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4098661e3e364e75945c10b462433269" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 46625 } }
I20260926 01:03:51.329473 25367 leader_election.cc:302] T 00000000000000000000000000000000 P 4098661e3e364e75945c10b462433269 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4098661e3e364e75945c10b462433269; no voters:
I20260926 01:03:51.329638 25367 leader_election.cc:288] T 00000000000000000000000000000000 P 4098661e3e364e75945c10b462433269 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:03:51.329656 25372 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4098661e3e364e75945c10b462433269 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:03:51.329778 25372 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4098661e3e364e75945c10b462433269 [term 1 LEADER]: Becoming Leader. State: Replica: 4098661e3e364e75945c10b462433269, State: Running, Role: LEADER
I20260926 01:03:51.329883 25372 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4098661e3e364e75945c10b462433269 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4098661e3e364e75945c10b462433269" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 46625 } }
I20260926 01:03:51.329909 25367 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4098661e3e364e75945c10b462433269 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:03:51.331193 25374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4098661e3e364e75945c10b462433269 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4098661e3e364e75945c10b462433269" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4098661e3e364e75945c10b462433269" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 46625 } } }
I20260926 01:03:51.331434 25373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4098661e3e364e75945c10b462433269 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4098661e3e364e75945c10b462433269. Latest consensus state: current_term: 1 leader_uuid: "4098661e3e364e75945c10b462433269" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4098661e3e364e75945c10b462433269" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 46625 } } }
Sep 26 01:03:51 dist-test-slave-7v10 krb5kdc[25284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 01:03:51.331694 25373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4098661e3e364e75945c10b462433269 [sys.catalog]: This master's current role is: LEADER
I20260926 01:03:51.331298 25374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4098661e3e364e75945c10b462433269 [sys.catalog]: This master's current role is: LEADER
W20260926 01:03:51.331905 25386 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4098661e3e364e75945c10b462433269: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:03:51.332000 25386 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:03:51.332195 25388 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:03:51.332398 25388 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:03:51.334272 25388 catalog_manager.cc:1347] Generated new cluster ID: db62e2e83fec4665b8d8d92ecc015ef1
I20260926 01:03:51.334340 25388 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:03:51.336093 25370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:51.325098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52945 (local address 127.3.75.62:46625)
0926 01:03:51.325259 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:51.325266 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:51.325290 (+ 24us) server_negotiation.cc:407] Connection header received
0926 01:03:51.325741 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:51.325748 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:51.325799 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:51.325907 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:51.326698 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:51.327603 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:51.328768 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:51.329207 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:51.329295 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:51.332257 (+ 2962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:51.332261 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:51.332261 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:03:51.332272 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:51.334305 (+ 2033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:51.334710 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:51.334713 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:51.334714 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:51.334820 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:51.335077 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:51.335078 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:51.335078 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:51.335404 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:51.335556 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:51.335672 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 01:03:51.335818 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":74,"threads_started":1}
I20260926 01:03:51.336982 3372 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:03:51.346356 25370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:51.338397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33047 (local address 127.3.75.62:46625)
0926 01:03:51.338433 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:51.338435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:51.339065 (+ 630us) server_negotiation.cc:407] Connection header received
0926 01:03:51.339154 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:51.339155 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:51.339164 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:51.339193 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:51.339894 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:51.340684 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:51.341790 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:51.342190 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:51.342228 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:51.343638 (+ 1410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:51.343639 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:51.343639 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:03:51.343645 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:51.345207 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:51.345581 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:51.345582 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:51.345582 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:51.345630 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:51.345806 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:51.345807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:51.345807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:51.345920 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:51.345987 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:51.346190 (+ 203us) server_negotiation.cc:299] Negotiation successful
0926 01:03:51.346262 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 01:03:51.367420 25388 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:03:51.368002 25388 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:03:51.373785 25388 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4098661e3e364e75945c10b462433269: Generated new TSK 0
I20260926 01:03:51.373994 25388 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:03:51.380287 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 25297
2026-09-26T01:03:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1323 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384372073861-3372-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:03:51 dist-test-slave-7v10 krb5kdc[25398](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:03:51 dist-test-slave-7v10 krb5kdc[25398](info): set up 2 sockets
Sep 26 01:03:51 dist-test-slave-7v10 krb5kdc[25398](info): commencing operation
krb5kdc: starting...
W20260926 01:03:52.410039 3372 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Sep 26 01:03:52 dist-test-slave-7v10 krb5kdc[25398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:03:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:03:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.75.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab.
Entry for principal kudu/127.3.75.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab.
I20260926 01:03:52.520999 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384372073861-3372-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384372073861-3372-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384372073861-3372-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:41239
--webserver_interface=127.3.75.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:37501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.75.62:41239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:03:52.599550 25411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:03:52.599772 25411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:03:52.599808 25411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:03:52.600950 25411 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:03:52.601017 25411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:03:52.601042 25411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:03:52.601071 25411 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:03:52.601104 25411 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:03:52.602541 25411 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:37501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384372073861-3372-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384372073861-3372-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.75.62:41239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:41239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384372073861-3372-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:03:52.602803 25411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:03:52.603045 25411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:03:52.607890 25417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:03:52.608018 25411 server_base.cc:1034] running on GCE node
W20260926 01:03:52.607890 25419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:52.608047 25416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:03:52.608314 25411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:03:52.608657 25411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:03:52.609817 25411 hybrid_clock.cc:648] HybridClock initialized: now 1790384632609809 us; error 28 us; skew 500 ppm
Sep 26 01:03:52 dist-test-slave-7v10 krb5kdc[25398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384632, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:03:52.612190 25411 init.cc:377] Logged in from keytab as kudu/127.3.75.62@KRBTEST.COM (short username kudu)
I20260926 01:03:52.612599 25411 webserver.cc:460] Webserver started at http://127.3.75.62:43591/ using document root <none> and password file <none>
I20260926 01:03:52.612831 25411 fs_manager.cc:359] Metadata directory not provided
I20260926 01:03:52.612896 25411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:03:52.613039 25411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:03:52.614027 25411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384372073861-3372-0/minicluster-data/master-0/data/instance:
uuid: "e136b3555e2c491f93e35ff9455ea55d"
format_stamp: "Formatted at 2026-09-26 01:03:52 on dist-test-slave-7v10"
server_key: "18dcb40b85a299f96a3e8969cee539c9"
server_key_iv: "6edba095d72146901feab02b82bec60b"
server_key_version: "encryptionkey@0"
I20260926 01:03:52.614372 25411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384372073861-3372-0/minicluster-data/master-0/wal/instance:
uuid: "e136b3555e2c491f93e35ff9455ea55d"
format_stamp: "Formatted at 2026-09-26 01:03:52 on dist-test-slave-7v10"
server_key: "18dcb40b85a299f96a3e8969cee539c9"
server_key_iv: "6edba095d72146901feab02b82bec60b"
server_key_version: "encryptionkey@0"
I20260926 01:03:52.615760 25411 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 01:03:52.616433 25426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:52.616598 25411 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 01:03:52.616711 25411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384372073861-3372-0/minicluster-data/master-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384372073861-3372-0/minicluster-data/master-0/wal
uuid: "e136b3555e2c491f93e35ff9455ea55d"
format_stamp: "Formatted at 2026-09-26 01:03:52 on dist-test-slave-7v10"
server_key: "18dcb40b85a299f96a3e8969cee539c9"
server_key_iv: "6edba095d72146901feab02b82bec60b"
server_key_version: "encryptionkey@0"
I20260926 01:03:52.616784 25411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384372073861-3372-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384372073861-3372-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384372073861-3372-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:03:52.641670 25411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:03:52.643677 25411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:03:52.643796 25411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:03:52.647976 25411 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.62:41239
I20260926 01:03:52.648051 25479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.62:41239 every 8 connection(s)
I20260926 01:03:52.648370 25411 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
I20260926 01:03:52.649223 25480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:03:52.651464 25480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e136b3555e2c491f93e35ff9455ea55d: Bootstrap starting.
I20260926 01:03:52.652009 25480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e136b3555e2c491f93e35ff9455ea55d: Neither blocks nor log segments found. Creating new log.
I20260926 01:03:52.652313 25480 log.cc:824] T 00000000000000000000000000000000 P e136b3555e2c491f93e35ff9455ea55d: Log is configured to *not* fsync() on all Append() calls
I20260926 01:03:52.652961 25480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e136b3555e2c491f93e35ff9455ea55d: No bootstrap required, opened a new log
I20260926 01:03:52.654438 25480 raft_consensus.cc:348] T 00000000000000000000000000000000 P e136b3555e2c491f93e35ff9455ea55d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e136b3555e2c491f93e35ff9455ea55d" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 41239 } }
I20260926 01:03:52.654593 25480 raft_consensus.cc:374] T 00000000000000000000000000000000 P e136b3555e2c491f93e35ff9455ea55d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:03:52.654639 25480 raft_consensus.cc:729] T 00000000000000000000000000000000 P e136b3555e2c491f93e35ff9455ea55d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e136b3555e2c491f93e35ff9455ea55d, State: Initialized, Role: FOLLOWER
I20260926 01:03:52.654775 25480 consensus_queue.cc:260] T 00000000000000000000000000000000 P e136b3555e2c491f93e35ff9455ea55d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e136b3555e2c491f93e35ff9455ea55d" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 41239 } }
I20260926 01:03:52.654853 25480 raft_consensus.cc:388] T 00000000000000000000000000000000 P e136b3555e2c491f93e35ff9455ea55d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:03:52.654922 25480 raft_consensus.cc:482] T 00000000000000000000000000000000 P e136b3555e2c491f93e35ff9455ea55d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:03:52.654994 25480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e136b3555e2c491f93e35ff9455ea55d [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:03:52.655587 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 25411
I20260926 01:03:52.655751 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384372073861-3372-0/minicluster-data/master-0/wal/instance
I20260926 01:03:52.655962 3372 external_mini_cluster.cc:1428] Setting key 32f69e21af88b3d34014a343e4cf13e3
I20260926 01:03:52.655999 25480 raft_consensus.cc:504] T 00000000000000000000000000000000 P e136b3555e2c491f93e35ff9455ea55d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e136b3555e2c491f93e35ff9455ea55d" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 41239 } }
I20260926 01:03:52.656107 25480 leader_election.cc:302] T 00000000000000000000000000000000 P e136b3555e2c491f93e35ff9455ea55d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e136b3555e2c491f93e35ff9455ea55d; no voters:
I20260926 01:03:52.656303 25480 leader_election.cc:288] T 00000000000000000000000000000000 P e136b3555e2c491f93e35ff9455ea55d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:03:52.656327 25483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e136b3555e2c491f93e35ff9455ea55d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:03:52.656783 25483 raft_consensus.cc:686] T 00000000000000000000000000000000 P e136b3555e2c491f93e35ff9455ea55d [term 1 LEADER]: Becoming Leader. State: Replica: e136b3555e2c491f93e35ff9455ea55d, State: Running, Role: LEADER
I20260926 01:03:52.656847 25480 sys_catalog.cc:564] T 00000000000000000000000000000000 P e136b3555e2c491f93e35ff9455ea55d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:03:52.656869 25483 consensus_queue.cc:237] T 00000000000000000000000000000000 P e136b3555e2c491f93e35ff9455ea55d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e136b3555e2c491f93e35ff9455ea55d" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 41239 } }
I20260926 01:03:52.657372 25486 sys_catalog.cc:455] T 00000000000000000000000000000000 P e136b3555e2c491f93e35ff9455ea55d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e136b3555e2c491f93e35ff9455ea55d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e136b3555e2c491f93e35ff9455ea55d" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 41239 } } }
I20260926 01:03:52.657542 25486 sys_catalog.cc:458] T 00000000000000000000000000000000 P e136b3555e2c491f93e35ff9455ea55d [sys.catalog]: This master's current role is: LEADER
I20260926 01:03:52.658077 25487 sys_catalog.cc:455] T 00000000000000000000000000000000 P e136b3555e2c491f93e35ff9455ea55d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e136b3555e2c491f93e35ff9455ea55d. Latest consensus state: current_term: 1 leader_uuid: "e136b3555e2c491f93e35ff9455ea55d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e136b3555e2c491f93e35ff9455ea55d" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 41239 } } }
I20260926 01:03:52.658250 25487 sys_catalog.cc:458] T 00000000000000000000000000000000 P e136b3555e2c491f93e35ff9455ea55d [sys.catalog]: This master's current role is: LEADER
W20260926 01:03:52.658670 25501 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e136b3555e2c491f93e35ff9455ea55d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:03:52.658731 25501 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:03:52.658779 25494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:03:52.658962 25494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:03:52.660291 25494 catalog_manager.cc:1347] Generated new cluster ID: 4031778005844e53b3b2d7eb2f6d485a
I20260926 01:03:52.660343 25494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:03:52.662243 25485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:52.656445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34281 (local address 127.3.75.62:41239)
0926 01:03:52.656681 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:52.656691 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:52.656755 (+ 64us) server_negotiation.cc:407] Connection header received
0926 01:03:52.656874 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:52.656881 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:52.656936 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:52.657093 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:52.657315 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:52.658863 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:52.659991 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:52.660584 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:52.661016 (+ 432us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:52.661403 (+ 387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:52.661407 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:52.661407 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:03:52.661632 (+ 225us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:52.661668 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:03:52.661696 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:52.661941 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:52.661973 (+ 32us) server_negotiation.cc:299] Negotiation successful
0926 01:03:52.662037 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":95,"threads_started":1}
I20260926 01:03:52.663131 3372 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:03:52.667897 25485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:52.664638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33463 (local address 127.3.75.62:41239)
0926 01:03:52.664670 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:52.664672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:52.664681 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:03:52.664746 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:52.664747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:52.664755 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:52.664803 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:52.664924 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:52.665600 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:52.666609 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:52.666919 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:52.666986 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:52.667583 (+ 597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:52.667585 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:52.667585 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:03:52.667589 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:52.667603 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:03:52.667607 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:52.667658 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:52.667728 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 01:03:52.667749 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 01:03:52.696153 25494 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:03:52.696894 25494 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:03:52.705039 25494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e136b3555e2c491f93e35ff9455ea55d: Generated new TSK 0
I20260926 01:03:52.705230 25494 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:03:52.712627 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 25411
2026-09-26T01:03:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1331 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384372073861-3372-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:03:52 dist-test-slave-7v10 krb5kdc[25512](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:03:52 dist-test-slave-7v10 krb5kdc[25512](info): set up 2 sockets
Sep 26 01:03:52 dist-test-slave-7v10 krb5kdc[25512](info): commencing operation
krb5kdc: starting...
W20260926 01:03:53.749975 3372 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Sep 26 01:03:53 dist-test-slave-7v10 krb5kdc[25512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:03:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:03:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.75.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab.
Entry for principal kudu/127.3.75.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab.
I20260926 01:03:53.864076 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384372073861-3372-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384372073861-3372-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384372073861-3372-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:35285
--webserver_interface=127.3.75.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:41261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.75.62:35285
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:03:53.943725 25525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:03:53.943967 25525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:03:53.944005 25525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:03:53.945137 25525 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:03:53.945216 25525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:03:53.945240 25525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:03:53.945261 25525 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:03:53.945291 25525 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:03:53.946748 25525 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:41261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384372073861-3372-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384372073861-3372-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.75.62:35285
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:35285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384372073861-3372-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:03:53.947034 25525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:03:53.947255 25525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:03:53.951895 25531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:53.951915 25530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:53.951900 25533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:03:53.952174 25525 server_base.cc:1034] running on GCE node
I20260926 01:03:53.952401 25525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:03:53.952689 25525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:03:53.953838 25525 hybrid_clock.cc:648] HybridClock initialized: now 1790384633953814 us; error 38 us; skew 500 ppm
Sep 26 01:03:53 dist-test-slave-7v10 krb5kdc[25512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384633, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:03:53.960175 25525 init.cc:377] Logged in from keytab as kudu/127.3.75.62@KRBTEST.COM (short username kudu)
I20260926 01:03:53.960618 25525 webserver.cc:460] Webserver started at http://127.3.75.62:37671/ using document root <none> and password file <none>
I20260926 01:03:53.960837 25525 fs_manager.cc:359] Metadata directory not provided
I20260926 01:03:53.960882 25525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:03:53.961017 25525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:03:53.962023 25525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384372073861-3372-0/minicluster-data/master-0/data/instance:
uuid: "f80c119330614ed09c07804201bfdea3"
format_stamp: "Formatted at 2026-09-26 01:03:53 on dist-test-slave-7v10"
server_key: "f8e258cad72031753fc926371ab25c21"
server_key_iv: "f6a3426305829b34774f6ed87e181365"
server_key_version: "encryptionkey@0"
I20260926 01:03:53.962373 25525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384372073861-3372-0/minicluster-data/master-0/wal/instance:
uuid: "f80c119330614ed09c07804201bfdea3"
format_stamp: "Formatted at 2026-09-26 01:03:53 on dist-test-slave-7v10"
server_key: "f8e258cad72031753fc926371ab25c21"
server_key_iv: "f6a3426305829b34774f6ed87e181365"
server_key_version: "encryptionkey@0"
I20260926 01:03:53.963752 25525 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 01:03:53.964489 25540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:53.964689 25525 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 01:03:53.964789 25525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384372073861-3372-0/minicluster-data/master-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384372073861-3372-0/minicluster-data/master-0/wal
uuid: "f80c119330614ed09c07804201bfdea3"
format_stamp: "Formatted at 2026-09-26 01:03:53 on dist-test-slave-7v10"
server_key: "f8e258cad72031753fc926371ab25c21"
server_key_iv: "f6a3426305829b34774f6ed87e181365"
server_key_version: "encryptionkey@0"
I20260926 01:03:53.964861 25525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384372073861-3372-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384372073861-3372-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384372073861-3372-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:03:53.968214 25525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:03:53.970220 25525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:03:53.970355 25525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:03:53.974146 25525 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.62:35285
I20260926 01:03:53.974176 25592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.62:35285 every 8 connection(s)
I20260926 01:03:53.974494 25525 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
I20260926 01:03:53.975477 25593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:03:53.977640 25593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f80c119330614ed09c07804201bfdea3: Bootstrap starting.
I20260926 01:03:53.978176 25593 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f80c119330614ed09c07804201bfdea3: Neither blocks nor log segments found. Creating new log.
I20260926 01:03:53.978432 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 25525
I20260926 01:03:53.978554 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384372073861-3372-0/minicluster-data/master-0/wal/instance
I20260926 01:03:53.978551 25593 log.cc:824] T 00000000000000000000000000000000 P f80c119330614ed09c07804201bfdea3: Log is configured to *not* fsync() on all Append() calls
I20260926 01:03:53.978747 3372 external_mini_cluster.cc:1428] Setting key d2c872e0fd0a1b5f15e30c1d3098760b
I20260926 01:03:53.979393 25593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f80c119330614ed09c07804201bfdea3: No bootstrap required, opened a new log
I20260926 01:03:53.981500 25593 raft_consensus.cc:348] T 00000000000000000000000000000000 P f80c119330614ed09c07804201bfdea3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f80c119330614ed09c07804201bfdea3" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 35285 } }
I20260926 01:03:53.981704 25593 raft_consensus.cc:374] T 00000000000000000000000000000000 P f80c119330614ed09c07804201bfdea3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:03:53.981768 25593 raft_consensus.cc:729] T 00000000000000000000000000000000 P f80c119330614ed09c07804201bfdea3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f80c119330614ed09c07804201bfdea3, State: Initialized, Role: FOLLOWER
I20260926 01:03:53.981887 25593 consensus_queue.cc:260] T 00000000000000000000000000000000 P f80c119330614ed09c07804201bfdea3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f80c119330614ed09c07804201bfdea3" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 35285 } }
I20260926 01:03:53.981987 25593 raft_consensus.cc:388] T 00000000000000000000000000000000 P f80c119330614ed09c07804201bfdea3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:03:53.982019 25597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:53.979314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38991 (local address 127.3.75.62:35285)
0926 01:03:53.979515 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:53.979526 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:53.979573 (+ 47us) server_negotiation.cc:407] Connection header received
0926 01:03:53.980605 (+ 1032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:53.980613 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:53.980647 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:53.980796 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:53.981407 (+ 611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:53.981412 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:53.981412 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:03:53.981417 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:53.981461 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:03:53.981474 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:53.981569 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:53.981600 (+ 31us) server_negotiation.cc:299] Negotiation successful
0926 01:03:53.981637 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":89,"threads_started":1}
I20260926 01:03:53.982033 25593 raft_consensus.cc:482] T 00000000000000000000000000000000 P f80c119330614ed09c07804201bfdea3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:03:53.982096 25593 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f80c119330614ed09c07804201bfdea3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:03:53.982796 25593 raft_consensus.cc:504] T 00000000000000000000000000000000 P f80c119330614ed09c07804201bfdea3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f80c119330614ed09c07804201bfdea3" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 35285 } }
I20260926 01:03:53.982926 25593 leader_election.cc:302] T 00000000000000000000000000000000 P f80c119330614ed09c07804201bfdea3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f80c119330614ed09c07804201bfdea3; no voters:
I20260926 01:03:53.982990 3372 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:03:53.983088 25593 leader_election.cc:288] T 00000000000000000000000000000000 P f80c119330614ed09c07804201bfdea3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:03:53.983223 25599 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f80c119330614ed09c07804201bfdea3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:03:53.983312 25593 sys_catalog.cc:564] T 00000000000000000000000000000000 P f80c119330614ed09c07804201bfdea3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:03:53.983350 25599 raft_consensus.cc:686] T 00000000000000000000000000000000 P f80c119330614ed09c07804201bfdea3 [term 1 LEADER]: Becoming Leader. State: Replica: f80c119330614ed09c07804201bfdea3, State: Running, Role: LEADER
I20260926 01:03:53.983489 25599 consensus_queue.cc:237] T 00000000000000000000000000000000 P f80c119330614ed09c07804201bfdea3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f80c119330614ed09c07804201bfdea3" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 35285 } }
I20260926 01:03:53.983820 25600 sys_catalog.cc:455] T 00000000000000000000000000000000 P f80c119330614ed09c07804201bfdea3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f80c119330614ed09c07804201bfdea3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f80c119330614ed09c07804201bfdea3" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 35285 } } }
I20260926 01:03:53.983968 25600 sys_catalog.cc:458] T 00000000000000000000000000000000 P f80c119330614ed09c07804201bfdea3 [sys.catalog]: This master's current role is: LEADER
I20260926 01:03:53.983807 25599 sys_catalog.cc:455] T 00000000000000000000000000000000 P f80c119330614ed09c07804201bfdea3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f80c119330614ed09c07804201bfdea3. Latest consensus state: current_term: 1 leader_uuid: "f80c119330614ed09c07804201bfdea3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f80c119330614ed09c07804201bfdea3" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 35285 } } }
I20260926 01:03:53.984236 25599 sys_catalog.cc:458] T 00000000000000000000000000000000 P f80c119330614ed09c07804201bfdea3 [sys.catalog]: This master's current role is: LEADER
I20260926 01:03:53.984546 25609 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:03:53.985092 25609 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:03:53.986557 25597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:53.984765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43351 (local address 127.3.75.62:35285)
0926 01:03:53.984799 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:53.984801 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:53.985674 (+ 873us) server_negotiation.cc:407] Connection header received
0926 01:03:53.985714 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:53.985715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:53.985720 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:53.985778 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:53.986331 (+ 553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:53.986332 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:53.986332 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:03:53.986336 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:53.986349 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:03:53.986353 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:53.986453 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:53.986486 (+ 33us) server_negotiation.cc:299] Negotiation successful
0926 01:03:53.986498 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 01:03:53.991122 25609 catalog_manager.cc:1347] Generated new cluster ID: 2fe6b17eb6e64e9f8f27a52416bc3456
I20260926 01:03:53.991174 25609 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:03:54.003489 25609 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:03:54.003688 25609 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:03:54.012261 25609 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f80c119330614ed09c07804201bfdea3: Generated new TSK 0
I20260926 01:03:54.012440 25609 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:03:54.015749 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 25525
2026-09-26T01:03:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1302 ms)
[----------] 18 tests from AuthTokenIssuingTest (23755 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:03:54 dist-test-slave-7v10 krb5kdc[25624](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:03:54 dist-test-slave-7v10 krb5kdc[25624](info): set up 2 sockets
Sep 26 01:03:54 dist-test-slave-7v10 krb5kdc[25624](info): commencing operation
krb5kdc: starting...
W20260926 01:03:55.062009 3372 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Sep 26 01:03:55 dist-test-slave-7v10 krb5kdc[25624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:03:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:03:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.75.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab.
Entry for principal kudu/127.3.75.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab.
I20260926 01:03:55.178251 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:44911
--webserver_interface=127.3.75.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:42133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.75.62:44911,127.3.75.61:39535,127.3.75.60:39741
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:03:55.259333 25637 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 01:03:55.259548 25637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:03:55.259565 25637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:03:55.259584 25637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:03:55.260641 25637 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:03:55.260686 25637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:03:55.260700 25637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:03:55.260713 25637 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:03:55.260725 25637 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:03:55.262113 25637 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:42133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.75.62:44911,127.3.75.61:39535,127.3.75.60:39741
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:44911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:03:55.262302 25637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:03:55.262472 25637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:03:55.267405 25645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:55.267370 25642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:55.267383 25643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:03:55.267411 25637 server_base.cc:1034] running on GCE node
I20260926 01:03:55.267876 25637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:03:55.268178 25637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:03:55.269325 25637 hybrid_clock.cc:648] HybridClock initialized: now 1790384635269313 us; error 30 us; skew 500 ppm
Sep 26 01:03:55 dist-test-slave-7v10 krb5kdc[25624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384635, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:03:55.271829 25637 init.cc:377] Logged in from keytab as kudu/127.3.75.62@KRBTEST.COM (short username kudu)
I20260926 01:03:55.272297 25637 webserver.cc:460] Webserver started at http://127.3.75.62:33193/ using document root <none> and password file <none>
I20260926 01:03:55.272537 25637 fs_manager.cc:359] Metadata directory not provided
I20260926 01:03:55.272605 25637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:03:55.272766 25637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:03:55.273825 25637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/minicluster-data/master-0/data/instance:
uuid: "47b52894892d4626b78c60d6675d4d21"
format_stamp: "Formatted at 2026-09-26 01:03:55 on dist-test-slave-7v10"
server_key: "a91075f8d67ea547d69e7fbecad05c54"
server_key_iv: "67ac168cee9a701ed0233b1c26f636ed"
server_key_version: "encryptionkey@0"
I20260926 01:03:55.274190 25637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/minicluster-data/master-0/wal/instance:
uuid: "47b52894892d4626b78c60d6675d4d21"
format_stamp: "Formatted at 2026-09-26 01:03:55 on dist-test-slave-7v10"
server_key: "a91075f8d67ea547d69e7fbecad05c54"
server_key_iv: "67ac168cee9a701ed0233b1c26f636ed"
server_key_version: "encryptionkey@0"
I20260926 01:03:55.275575 25637 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:03:55.276341 25652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:55.276502 25637 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:03:55.276608 25637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/minicluster-data/master-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/minicluster-data/master-0/wal
uuid: "47b52894892d4626b78c60d6675d4d21"
format_stamp: "Formatted at 2026-09-26 01:03:55 on dist-test-slave-7v10"
server_key: "a91075f8d67ea547d69e7fbecad05c54"
server_key_iv: "67ac168cee9a701ed0233b1c26f636ed"
server_key_version: "encryptionkey@0"
I20260926 01:03:55.276686 25637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:03:55.285889 25637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:03:55.288300 25637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:03:55.288468 25637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:03:55.292584 25637 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.62:44911
I20260926 01:03:55.292618 25705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.62:44911 every 8 connection(s)
I20260926 01:03:55.292980 25637 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
I20260926 01:03:55.293246 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 25637
I20260926 01:03:55.293358 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/minicluster-data/master-0/wal/instance
I20260926 01:03:55.293581 3372 external_mini_cluster.cc:1428] Setting key 833a5fd2fc548f6dfcb45594e0fa767e
I20260926 01:03:55.294111 25706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:03:55.295845 25706 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.75.62" port: 44911 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.75.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.61@KRBTEST.COM" created.
Sep 26 01:03:55 dist-test-slave-7v10 krb5kdc[25624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384635, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.62@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 01:03:55.308748 25710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:55.296671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43286 (local address 127.3.75.62:44911)
0926 01:03:55.296910 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:55.296915 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:55.296944 (+ 29us) server_negotiation.cc:407] Connection header received
0926 01:03:55.297508 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:55.297511 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:55.297537 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:55.297634 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:55.298397 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:55.299315 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:55.300670 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:55.300975 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:55.301048 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:55.305059 (+ 4011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:55.305065 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:55.305066 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:03:55.305073 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:55.307066 (+ 1993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:55.307462 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:55.307465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:55.307465 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:55.307550 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:55.307782 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:55.307783 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:55.307783 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:55.308004 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:55.308096 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:55.308281 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 01:03:55.308454 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":201,"thread_start_us":89,"threads_started":1}
I20260926 01:03:55.308746 25709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:55.296643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44911 (local address 127.0.0.1:43286)
0926 01:03:55.296823 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:55.296865 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:55.297344 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:55.297639 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:55.297642 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:55.298028 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:03:55.298279 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:55.298287 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:55.299416 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:55.299418 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:55.300575 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:55.300578 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:55.301083 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:55.301084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:55.301187 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:55.301724 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:03:55.301735 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:03:55.304951 (+ 3216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:03:55.307157 (+ 2206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:03:55.307163 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:03:55.307165 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:03:55.307365 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:03:55.307627 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:03:55.307628 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:03:55.307628 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:03:55.307716 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:03:55.308076 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:03:55.308081 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:03:55.308143 (+ 62us) client_negotiation.cc:769] Sending connection context
0926 01:03:55.308258 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 01:03:55.308500 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":102,"thread_start_us":83,"threads_started":1}
I20260926 01:03:55.309705 25706 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.75.61" port: 39535 } has no permanent_uuid. Determining permanent_uuid...
I20260926 01:03:55.310068 25709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:55.309933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.61:39535 (local address 127.0.0.1:51282)
0926 01:03:55.309985 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:55.310037 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.61:39535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 01:03:55.310240 25656 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.75.61:39535: connect: Connection refused (error 111)
I20260926 01:03:55.310685 25709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:55.310560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.61:39535 (local address 127.0.0.1:51296)
0926 01:03:55.310626 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:55.310655 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.61:39535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 01:03:55.310963 25706 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.75.61:39535: Network error: Client connection negotiation failed: client connection to 127.3.75.61:39535: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.61.keytab.
Entry for principal kudu/127.3.75.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.61.keytab.
I20260926 01:03:55.319918 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.61:39535
--webserver_interface=127.3.75.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.61.keytab
--principal=kudu/127.3.75.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:42133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.75.62:44911,127.3.75.61:39535,127.3.75.60:39741
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 01:03:55.360209 25706 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.75.61" port: 39535 } attempt: 1
I20260926 01:03:55.360778 25709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:55.360633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.61:39535 (local address 127.0.0.1:51306)
0926 01:03:55.360705 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:55.360751 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.61:39535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 01:03:55.361151 25709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:55.361069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.61:39535 (local address 127.0.0.1:51310)
0926 01:03:55.361110 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:55.361134 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.61:39535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 01:03:55.361274 25706 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.75.61:39535: Network error: Client connection negotiation failed: client connection to 127.3.75.61:39535: connect: Connection refused (error 111)
W20260926 01:03:55.404534 25713 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 01:03:55.404779 25713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:03:55.404807 25713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:03:55.404848 25713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:03:55.405966 25713 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:03:55.406039 25713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:03:55.406066 25713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:03:55.406098 25713 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:03:55.406126 25713 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:03:55.407572 25713 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:42133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.75.62:44911,127.3.75.61:39535,127.3.75.60:39741
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.61.keytab
--principal=kudu/127.3.75.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.61:39535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.75.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:03:55.407862 25713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:03:55.408097 25713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:03:55.412961 25719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:55.412993 25721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:55.413091 25718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:03:55.413103 25713 server_base.cc:1034] running on GCE node
I20260926 01:03:55.413407 25713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:03:55.413640 25713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:03:55.414779 25713 hybrid_clock.cc:648] HybridClock initialized: now 1790384635414768 us; error 26 us; skew 500 ppm
Sep 26 01:03:55 dist-test-slave-7v10 krb5kdc[25624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384635, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:03:55.417349 25713 init.cc:377] Logged in from keytab as kudu/127.3.75.61@KRBTEST.COM (short username kudu)
I20260926 01:03:55.417800 25713 webserver.cc:460] Webserver started at http://127.3.75.61:42147/ using document root <none> and password file <none>
I20260926 01:03:55.418045 25713 fs_manager.cc:359] Metadata directory not provided
I20260926 01:03:55.418088 25713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:03:55.418257 25713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:03:55.419356 25713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/minicluster-data/master-1/data/instance:
uuid: "ddaed7495f1a476ca21d0a231b8ea7ea"
format_stamp: "Formatted at 2026-09-26 01:03:55 on dist-test-slave-7v10"
server_key: "d069c7e80cd19ca1ce47781348cd76d5"
server_key_iv: "93288868399e70c9e947947e28609aed"
server_key_version: "encryptionkey@0"
I20260926 01:03:55.419708 25713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/minicluster-data/master-1/wal/instance:
uuid: "ddaed7495f1a476ca21d0a231b8ea7ea"
format_stamp: "Formatted at 2026-09-26 01:03:55 on dist-test-slave-7v10"
server_key: "d069c7e80cd19ca1ce47781348cd76d5"
server_key_iv: "93288868399e70c9e947947e28609aed"
server_key_version: "encryptionkey@0"
I20260926 01:03:55.420970 25713 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:03:55.421700 25728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:55.421896 25713 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:03:55.421986 25713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/minicluster-data/master-1/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/minicluster-data/master-1/wal
uuid: "ddaed7495f1a476ca21d0a231b8ea7ea"
format_stamp: "Formatted at 2026-09-26 01:03:55 on dist-test-slave-7v10"
server_key: "d069c7e80cd19ca1ce47781348cd76d5"
server_key_iv: "93288868399e70c9e947947e28609aed"
server_key_version: "encryptionkey@0"
I20260926 01:03:55.422057 25713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:03:55.429420 25706 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.75.61" port: 39535 } attempt: 2
I20260926 01:03:55.429811 25709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:55.429688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.61:39535 (local address 127.0.0.1:51318)
0926 01:03:55.429747 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:55.429783 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.61:39535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 01:03:55.430137 25709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:55.430040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.61:39535 (local address 127.0.0.1:51326)
0926 01:03:55.430086 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:55.430110 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.61:39535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 01:03:55.430229 25706 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.75.61:39535: Network error: Client connection negotiation failed: client connection to 127.3.75.61:39535: connect: Connection refused (error 111)
I20260926 01:03:55.446332 25713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:03:55.448838 25713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:03:55.449007 25713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:03:55.453110 25713 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.61:39535
I20260926 01:03:55.453145 25780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.61:39535 every 8 connection(s)
I20260926 01:03:55.453544 25713 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/minicluster-data/master-1/data/info.pb
I20260926 01:03:55.454485 25781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:03:55.454944 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 25713
I20260926 01:03:55.455057 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/minicluster-data/master-1/wal/instance
I20260926 01:03:55.455269 3372 external_mini_cluster.cc:1428] Setting key fa43edc226fbb68be46d523962e75cff
I20260926 01:03:55.456283 25781 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.75.62" port: 44911 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 01:03:55 dist-test-slave-7v10 krb5kdc[25624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384635, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.61@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.75.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.60@KRBTEST.COM" created.
I20260926 01:03:55.465988 25710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:55.457024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43292 (local address 127.3.75.62:44911)
0926 01:03:55.457094 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:55.457096 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:55.457339 (+ 243us) server_negotiation.cc:407] Connection header received
0926 01:03:55.457899 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:55.457901 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:55.457916 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:55.457985 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:55.458668 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:55.459512 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:55.460474 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:55.460805 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:55.460881 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:55.463208 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:55.463210 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:55.463210 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:03:55.463217 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:55.464521 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:55.465003 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:55.465003 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:55.465004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:55.465054 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:55.465289 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:55.465290 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:55.465290 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:55.465439 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:55.465509 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:55.465810 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 01:03:55.465898 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 01:03:55.466151 25784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:55.456992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44911 (local address 127.0.0.1:43292)
0926 01:03:55.457182 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:55.457231 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:55.457782 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:55.458043 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:55.458046 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:55.458302 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:03:55.458494 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:55.458499 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:55.459606 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:55.459607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:55.460385 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:55.460387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:55.460913 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:55.460914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:55.461007 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:55.461536 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:03:55.461545 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:03:55.463106 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:03:55.464594 (+ 1488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:03:55.464600 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:03:55.464602 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:03:55.464937 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:03:55.465112 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:03:55.465112 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:03:55.465113 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:03:55.465208 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:03:55.465515 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:03:55.465521 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:03:55.465665 (+ 144us) client_negotiation.cc:769] Sending connection context
0926 01:03:55.465772 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 01:03:55.465922 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":99,"threads_started":1}
I20260926 01:03:55.466776 25781 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.75.61" port: 39535 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 01:03:55 dist-test-slave-7v10 krb5kdc[25624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384635, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.61@KRBTEST.COM for kudu/127.3.75.61@KRBTEST.COM
I20260926 01:03:55.474327 25784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:55.467061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.61:39535 (local address 127.0.0.1:51338)
0926 01:03:55.467118 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:55.467131 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:55.467176 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:55.467419 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:55.467420 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:55.467578 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:03:55.467636 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:55.467638 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:55.468453 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:55.468454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:55.469338 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:55.469341 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:55.469800 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:55.469801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:55.469890 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:55.470239 (+ 349us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:03:55.470244 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:03:55.471568 (+ 1324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:03:55.473307 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:03:55.473308 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:03:55.473308 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:03:55.473470 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:03:55.473690 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:03:55.473691 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:03:55.473692 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:03:55.473751 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:03:55.474017 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:03:55.474018 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:03:55.474054 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 01:03:55.474120 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 01:03:55.474217 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 01:03:55.474411 25786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:55.467103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51338 (local address 127.3.75.61:39535)
0926 01:03:55.467273 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:55.467278 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:55.467296 (+ 18us) server_negotiation.cc:407] Connection header received
0926 01:03:55.467317 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:55.467319 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:55.467331 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:55.467400 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:55.467727 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:55.468365 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:55.469421 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:55.469720 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:55.469774 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:55.471667 (+ 1893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:55.471674 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:55.471674 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:03:55.471681 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:55.473186 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:55.473561 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:55.473564 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:55.473565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:55.473618 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:55.473810 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:55.473811 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:55.473812 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:55.473941 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:55.474025 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:55.474160 (+ 135us) server_negotiation.cc:299] Negotiation successful
0926 01:03:55.474284 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":142,"thread_start_us":115,"threads_started":1}
I20260926 01:03:55.475158 25781 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.75.60" port: 39741 } has no permanent_uuid. Determining permanent_uuid...
I20260926 01:03:55.475498 25784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:55.475381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.60:39741 (local address 127.0.0.1:40152)
0926 01:03:55.475428 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:55.475471 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.60:39741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 01:03:55.475625 25729 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.75.60:39741: connect: Connection refused (error 111)
I20260926 01:03:55.475984 25784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:55.475891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.60:39741 (local address 127.0.0.1:40166)
0926 01:03:55.475941 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:55.475964 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.60:39741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 01:03:55.476089 25781 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.75.60:39741: Network error: Client connection negotiation failed: client connection to 127.3.75.60:39741: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.60.keytab.
Entry for principal kudu/127.3.75.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.60.keytab.
I20260926 01:03:55.479902 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.60:39741
--webserver_interface=127.3.75.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.60.keytab
--principal=kudu/127.3.75.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:42133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.75.62:44911,127.3.75.61:39535,127.3.75.60:39741
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 01:03:55.521407 25706 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.75.61" port: 39535 } attempt: 3
I20260926 01:03:55.525285 25781 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.75.60" port: 39741 } attempt: 1
I20260926 01:03:55.525710 25784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:55.525571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.60:39741 (local address 127.0.0.1:40174)
0926 01:03:55.525636 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:55.525675 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.60:39741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
Sep 26 01:03:55 dist-test-slave-7v10 krb5kdc[25624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384635, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.62@KRBTEST.COM for kudu/127.3.75.61@KRBTEST.COM
I20260926 01:03:55.526139 25784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:55.526026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.60:39741 (local address 127.0.0.1:40186)
0926 01:03:55.526080 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:55.526111 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.60:39741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 01:03:55.526283 25781 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.75.60:39741: Network error: Client connection negotiation failed: client connection to 127.3.75.60:39741: connect: Connection refused (error 111)
I20260926 01:03:55.529244 25709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:55.521848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.61:39535 (local address 127.0.0.1:51344)
0926 01:03:55.521896 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:55.521910 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:55.521986 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:55.522148 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:55.522149 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:55.522320 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:03:55.522385 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:55.522387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:55.523170 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:55.523171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:55.523693 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:55.523695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:55.524086 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:55.524088 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:55.524190 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:55.524740 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:03:55.524744 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:03:55.526251 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:03:55.528026 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:03:55.528027 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:03:55.528027 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:03:55.528221 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:03:55.528473 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:03:55.528474 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:03:55.528474 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:03:55.528577 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:03:55.528854 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:03:55.528855 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:03:55.528919 (+ 64us) client_negotiation.cc:769] Sending connection context
0926 01:03:55.529034 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 01:03:55.529146 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260926 01:03:55.529243 25786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:55.521924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51344 (local address 127.3.75.61:39535)
0926 01:03:55.522003 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:55.522005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:55.522016 (+ 11us) server_negotiation.cc:407] Connection header received
0926 01:03:55.522047 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:55.522048 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:55.522059 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:55.522126 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:55.522481 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:55.523055 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:55.523785 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:55.524009 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:55.524089 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:55.526356 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:55.526358 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:55.526358 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:03:55.526365 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:55.527888 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:55.528317 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:55.528319 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:55.528319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:55.528388 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:55.528648 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:55.528649 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:55.528650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:55.528779 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:55.528863 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:55.529063 (+ 200us) server_negotiation.cc:299] Negotiation successful
0926 01:03:55.529147 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 01:03:55.529711 25706 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.75.60" port: 39741 } has no permanent_uuid. Determining permanent_uuid...
I20260926 01:03:55.530094 25709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:55.529967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.60:39741 (local address 127.0.0.1:40198)
0926 01:03:55.530031 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:55.530060 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.60:39741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 01:03:55.530443 25709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:55.530350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.60:39741 (local address 127.0.0.1:40214)
0926 01:03:55.530395 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:55.530420 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.60:39741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 01:03:55.530596 25706 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.75.60:39741: Network error: Client connection negotiation failed: client connection to 127.3.75.60:39741: connect: Connection refused (error 111)
W20260926 01:03:55.561532 25788 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 01:03:55.561815 25788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:03:55.561839 25788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
I20260926 01:03:55.561769 25706 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.75.60" port: 39741 } attempt: 1
W20260926 01:03:55.561867 25788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 01:03:55.562284 25709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:55.562128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.60:39741 (local address 127.0.0.1:40230)
0926 01:03:55.562201 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:55.562249 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.60:39741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 01:03:55.562664 25709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:55.562567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.60:39741 (local address 127.0.0.1:40238)
0926 01:03:55.562612 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:55.562641 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.60:39741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 01:03:55.562793 25706 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.75.60:39741: Network error: Client connection negotiation failed: client connection to 127.3.75.60:39741: connect: Connection refused (error 111)
W20260926 01:03:55.563454 25788 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:03:55.563498 25788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:03:55.563511 25788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:03:55.563556 25788 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:03:55.563583 25788 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:03:55.565162 25788 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:42133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.75.62:44911,127.3.75.61:39535,127.3.75.60:39741
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.60.keytab
--principal=kudu/127.3.75.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.60:39741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.75.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:03:55.565424 25788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:03:55.565683 25788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:03:55.570452 25794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:55.570490 25793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:55.570472 25796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:03:55.570726 25788 server_base.cc:1034] running on GCE node
I20260926 01:03:55.571004 25788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:03:55.571338 25788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:03:55.572502 25788 hybrid_clock.cc:648] HybridClock initialized: now 1790384635572486 us; error 47 us; skew 500 ppm
Sep 26 01:03:55 dist-test-slave-7v10 krb5kdc[25624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384635, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:03:55.574981 25788 init.cc:377] Logged in from keytab as kudu/127.3.75.60@KRBTEST.COM (short username kudu)
I20260926 01:03:55.575394 25788 webserver.cc:460] Webserver started at http://127.3.75.60:38641/ using document root <none> and password file <none>
I20260926 01:03:55.575621 25788 fs_manager.cc:359] Metadata directory not provided
I20260926 01:03:55.575697 25788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:03:55.575831 25788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:03:55.576910 25788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/minicluster-data/master-2/data/instance:
uuid: "0131d0575c9f4ae09eef7919b417e037"
format_stamp: "Formatted at 2026-09-26 01:03:55 on dist-test-slave-7v10"
server_key: "dd2bec7e0fcdacf66f3ca11b98210e98"
server_key_iv: "ce8b9346a0232cbe7adffabfaaddbfca"
server_key_version: "encryptionkey@0"
I20260926 01:03:55.577277 25788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/minicluster-data/master-2/wal/instance:
uuid: "0131d0575c9f4ae09eef7919b417e037"
format_stamp: "Formatted at 2026-09-26 01:03:55 on dist-test-slave-7v10"
server_key: "dd2bec7e0fcdacf66f3ca11b98210e98"
server_key_iv: "ce8b9346a0232cbe7adffabfaaddbfca"
server_key_version: "encryptionkey@0"
I20260926 01:03:55.578598 25788 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:03:55.579439 25804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:55.579623 25788 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:55.579751 25788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/minicluster-data/master-2/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/minicluster-data/master-2/wal
uuid: "0131d0575c9f4ae09eef7919b417e037"
format_stamp: "Formatted at 2026-09-26 01:03:55 on dist-test-slave-7v10"
server_key: "dd2bec7e0fcdacf66f3ca11b98210e98"
server_key_iv: "ce8b9346a0232cbe7adffabfaaddbfca"
server_key_version: "encryptionkey@0"
I20260926 01:03:55.579838 25788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:03:55.592905 25788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:03:55.594439 25781 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.75.60" port: 39741 } attempt: 2
I20260926 01:03:55.594928 25784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:55.594753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.60:39741 (local address 127.0.0.1:40244)
0926 01:03:55.594829 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:55.594892 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.60:39741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 01:03:55.595032 25788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:03:55.595170 25788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:03:55.595293 25784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:55.595184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.60:39741 (local address 127.0.0.1:40254)
0926 01:03:55.595237 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:55.595271 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.60:39741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 01:03:55.595404 25781 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.75.60:39741: Network error: Client connection negotiation failed: client connection to 127.3.75.60:39741: connect: Connection refused (error 111)
I20260926 01:03:55.599089 25788 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.60:39741
I20260926 01:03:55.599140 25856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.60:39741 every 8 connection(s)
I20260926 01:03:55.599476 25788 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/minicluster-data/master-2/data/info.pb
I20260926 01:03:55.600452 25857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:03:55.601979 25857 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.75.62" port: 44911 } has no permanent_uuid. Determining permanent_uuid...
I20260926 01:03:55.604406 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 25788
I20260926 01:03:55.604487 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384372073861-3372-0/minicluster-data/master-2/wal/instance
I20260926 01:03:55.604667 3372 external_mini_cluster.cc:1428] Setting key f701c65425e786dc45168b31b20b24b2
Sep 26 01:03:55 dist-test-slave-7v10 krb5kdc[25624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384635, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.60@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
Sep 26 01:03:55 dist-test-slave-7v10 krb5kdc[25624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.75.60@KRBTEST.COM
I20260926 01:03:55.611392 25710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:55.602668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43296 (local address 127.3.75.62:44911)
0926 01:03:55.602762 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:55.602764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:55.602960 (+ 196us) server_negotiation.cc:407] Connection header received
0926 01:03:55.603457 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:55.603459 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:55.603470 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:55.603527 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:55.604144 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:55.604717 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:55.605957 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:55.606171 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:55.606252 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:55.608461 (+ 2209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:55.608463 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:55.608463 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:03:55.608472 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:55.609933 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:55.610403 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:55.610404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:55.610405 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:55.610467 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:55.610721 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:55.610722 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:55.610723 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:55.610839 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:55.610944 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:55.611226 (+ 282us) server_negotiation.cc:299] Negotiation successful
0926 01:03:55.611302 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 01:03:55.611569 25859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:55.602619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:44911 (local address 127.0.0.1:43296)
0926 01:03:55.602804 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:55.602839 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:55.603348 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:55.603604 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:55.603608 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:55.603849 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:03:55.604039 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:55.604044 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:55.604794 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:55.604795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:55.605870 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:55.605872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:55.606292 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:55.606293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:55.606404 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:55.606945 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:03:55.606953 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:03:55.608370 (+ 1417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:03:55.610010 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:03:55.610018 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:03:55.610020 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:03:55.610326 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:03:55.610535 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:03:55.610536 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:03:55.610536 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:03:55.610651 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:03:55.610938 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:03:55.610945 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:03:55.611081 (+ 136us) client_negotiation.cc:769] Sending connection context
0926 01:03:55.611202 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 01:03:55.611348 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":78,"threads_started":1}
I20260926 01:03:55.612054 25857 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.75.61" port: 39535 } has no permanent_uuid. Determining permanent_uuid...
I20260926 01:03:55.613289 25861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:55.605071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40268 (local address 127.3.75.60:39741)
0926 01:03:55.605293 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:55.605297 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:55.605317 (+ 20us) server_negotiation.cc:407] Connection header received
0926 01:03:55.605339 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:55.605341 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:55.605359 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:55.605439 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:55.606032 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:55.606850 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:55.607621 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:55.607925 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:55.608009 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:55.610235 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:55.610241 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:55.610241 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:03:55.610247 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:55.612031 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:55.612452 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:55.612457 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:55.612457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:55.612513 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:55.612718 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:55.612719 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:55.612720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:55.612861 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:55.613020 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:55.613100 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 01:03:55.613177 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":139,"threads_started":1}
Sep 26 01:03:55 dist-test-slave-7v10 krb5kdc[25624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384635, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.60@KRBTEST.COM for kudu/127.3.75.61@KRBTEST.COM
Sep 26 01:03:55 dist-test-slave-7v10 krb5kdc[25624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 01:03:55.618791 25859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:55.612296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.61:39535 (local address 127.0.0.1:51356)
0926 01:03:55.612343 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:55.612352 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:55.612408 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:55.612531 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:55.612532 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:55.612658 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:03:55.612684 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:55.612685 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:55.613290 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:55.613291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:55.614053 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:55.614056 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:55.614554 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:55.614554 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:55.614598 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:55.615011 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:03:55.615016 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:03:55.616316 (+ 1300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:03:55.617766 (+ 1450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:03:55.617767 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:03:55.617767 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:03:55.617985 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:03:55.618204 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:03:55.618205 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:03:55.618205 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:03:55.618255 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:03:55.618525 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:03:55.618526 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:03:55.618559 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 01:03:55.618608 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 01:03:55.618717 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 01:03:55.618844 25786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:55.612368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51356 (local address 127.3.75.61:39535)
0926 01:03:55.612437 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:55.612439 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:55.612448 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:03:55.612471 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:55.612472 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:55.612481 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:55.612531 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:55.612758 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:55.613212 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:55.614145 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:55.614476 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:55.614556 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:55.616392 (+ 1836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:55.616393 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:55.616394 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:03:55.616401 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:55.617684 (+ 1283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:55.618066 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:55.618067 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:55.618067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:55.618132 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:55.618318 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:55.618319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:55.618320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:55.618462 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:55.618531 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:55.618651 (+ 120us) server_negotiation.cc:299] Negotiation successful
0926 01:03:55.618735 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 01:03:55.619218 25857 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.75.60" port: 39741 } has no permanent_uuid. Determining permanent_uuid...
I20260926 01:03:55.621745 25710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:55.614086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43300 (local address 127.3.75.62:44911)
0926 01:03:55.614129 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:55.614130 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:55.614137 (+ 7us) server_negotiation.cc:407] Connection header received
0926 01:03:55.614156 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:55.614156 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:55.614164 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:55.614217 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:55.614762 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:55.615278 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:55.616276 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:55.616555 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:55.616631 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:55.619181 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:55.619183 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:55.619183 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:03:55.619197 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:55.620683 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:55.620992 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:55.620993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:55.620994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:55.621052 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:55.621232 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:55.621233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:55.621233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:55.621389 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:55.621519 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:55.621593 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 01:03:55.621648 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
Sep 26 01:03:55 dist-test-slave-7v10 krb5kdc[25624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384635, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.60@KRBTEST.COM for kudu/127.3.75.60@KRBTEST.COM
I20260926 01:03:55.626044 25861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:55.619545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40274 (local address 127.3.75.60:39741)
0926 01:03:55.619582 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:55.619584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:55.619592 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:03:55.619639 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:55.619640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:55.619656 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:55.619713 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:55.619980 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:55.620494 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:55.621363 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:55.621664 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:55.621726 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:55.623579 (+ 1853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:55.623580 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:55.623581 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:03:55.623588 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:55.624970 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:55.625302 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:55.625303 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:55.625303 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:55.625350 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:55.625546 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:55.625547 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:55.625547 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:55.625652 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:55.625725 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:55.625848 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 01:03:55.625925 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 01:03:55.626109 25859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:55.619486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.60:39741 (local address 127.0.0.1:40274)
0926 01:03:55.619538 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:55.619549 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:55.619591 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:55.619723 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:55.619724 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:55.619868 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:03:55.619907 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:55.619909 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:55.620575 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:55.620576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:55.621291 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:55.621293 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:55.621739 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:55.621739 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:55.621797 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:55.622228 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:03:55.622233 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:03:55.623500 (+ 1267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:03:55.625057 (+ 1557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:03:55.625058 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:03:55.625058 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:03:55.625220 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:03:55.625422 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:03:55.625423 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:03:55.625423 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:03:55.625475 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:03:55.625718 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:03:55.625718 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:03:55.625755 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 01:03:55.625823 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 01:03:55.625975 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":61}
Sep 26 01:03:55 dist-test-slave-7v10 krb5kdc[25624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.75.61@KRBTEST.COM
I20260926 01:03:55.627882 25857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0131d0575c9f4ae09eef7919b417e037: Bootstrap starting.
I20260926 01:03:55.628770 25857 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0131d0575c9f4ae09eef7919b417e037: Neither blocks nor log segments found. Creating new log.
I20260926 01:03:55.629070 25857 log.cc:824] T 00000000000000000000000000000000 P 0131d0575c9f4ae09eef7919b417e037: Log is configured to *not* fsync() on all Append() calls
I20260926 01:03:55.629791 25786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:55.622349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51366 (local address 127.3.75.61:39535)
0926 01:03:55.622393 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:55.622395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:55.622403 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:03:55.622423 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:55.622424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:55.622432 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:55.622486 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:55.623101 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:55.623580 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:55.624534 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:55.624767 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:55.624841 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:55.627263 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:55.627265 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:55.627265 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:03:55.627272 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:55.628587 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:55.628989 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:55.628990 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:55.628991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:55.629049 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:55.629237 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:55.629238 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:55.629239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:55.629373 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:55.629495 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:55.629629 (+ 134us) server_negotiation.cc:299] Negotiation successful
0926 01:03:55.629706 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 01:03:55.629937 25857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0131d0575c9f4ae09eef7919b417e037: No bootstrap required, opened a new log
I20260926 01:03:55.631270 3372 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:03:55.631748 25857 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0131d0575c9f4ae09eef7919b417e037 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47b52894892d4626b78c60d6675d4d21" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 44911 } } peers { permanent_uuid: "ddaed7495f1a476ca21d0a231b8ea7ea" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 39535 } } peers { permanent_uuid: "0131d0575c9f4ae09eef7919b417e037" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 39741 } }
I20260926 01:03:55.631867 25857 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0131d0575c9f4ae09eef7919b417e037 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0131d0575c9f4ae09eef7919b417e037, State: Initialized, Role: FOLLOWER
I20260926 01:03:55.631949 25857 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0131d0575c9f4ae09eef7919b417e037 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47b52894892d4626b78c60d6675d4d21" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 44911 } } peers { permanent_uuid: "ddaed7495f1a476ca21d0a231b8ea7ea" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 39535 } } peers { permanent_uuid: "0131d0575c9f4ae09eef7919b417e037" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 39741 } }
I20260926 01:03:55.632246 25857 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0131d0575c9f4ae09eef7919b417e037 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:03:55.632201 25863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0131d0575c9f4ae09eef7919b417e037 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47b52894892d4626b78c60d6675d4d21" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 44911 } } peers { permanent_uuid: "ddaed7495f1a476ca21d0a231b8ea7ea" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 39535 } } peers { permanent_uuid: "0131d0575c9f4ae09eef7919b417e037" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 39741 } } }
I20260926 01:03:55.632308 25863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0131d0575c9f4ae09eef7919b417e037 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:03:55.632465 25680 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "47b52894892d4626b78c60d6675d4d21"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43300
W20260926 01:03:55.634027 25874 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0131d0575c9f4ae09eef7919b417e037: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:03:55.634075 25874 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:03:55.634310 25788 master_runner.cc:186] Error getting master registration for 127.3.75.62:44911: OK, instance_id { permanent_uuid: "47b52894892d4626b78c60d6675d4d21" instance_seqno: 1790384635291453 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 01:03:55.636369 25788 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 01:03:55.637950 25706 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.75.60" port: 39741 } attempt: 2
Sep 26 01:03:55 dist-test-slave-7v10 krb5kdc[25624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384635, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.62@KRBTEST.COM for kudu/127.3.75.60@KRBTEST.COM
I20260926 01:03:55.646386 25709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:55.638218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.60:39741 (local address 127.0.0.1:40288)
0926 01:03:55.638278 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:55.638289 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:55.638327 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:55.638574 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:55.638576 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:55.638733 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:03:55.638770 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:55.638771 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:55.639764 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:55.639766 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:55.640568 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:55.640570 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:55.641075 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:55.641076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:55.641129 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:55.641604 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:03:55.641610 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:03:55.643250 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:03:55.645180 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:03:55.645181 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:03:55.645181 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:03:55.645410 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:03:55.645623 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:03:55.645640 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:03:55.645640 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:03:55.645699 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:03:55.645995 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:03:55.645996 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:03:55.646043 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 01:03:55.646164 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 01:03:55.646275 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 01:03:55.646584 25861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:55.638405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40288 (local address 127.3.75.60:39741)
0926 01:03:55.638459 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:55.638460 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:55.638468 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:03:55.638487 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:55.638487 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:55.638496 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:55.638569 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:55.638890 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:55.639672 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:55.640655 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:55.640966 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:55.641022 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:55.643346 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:55.643347 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:55.643348 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:03:55.643355 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:55.645096 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:55.645495 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:55.645496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:55.645496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:55.645571 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:55.645749 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:55.645750 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:55.645751 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:55.645935 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:55.645997 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:55.646407 (+ 410us) server_negotiation.cc:299] Negotiation successful
0926 01:03:55.646487 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 01:03:55.648146 25706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47b52894892d4626b78c60d6675d4d21: Bootstrap starting.
I20260926 01:03:55.648854 25706 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47b52894892d4626b78c60d6675d4d21: Neither blocks nor log segments found. Creating new log.
I20260926 01:03:55.649160 25706 log.cc:824] T 00000000000000000000000000000000 P 47b52894892d4626b78c60d6675d4d21: Log is configured to *not* fsync() on all Append() calls
I20260926 01:03:55.649915 25706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47b52894892d4626b78c60d6675d4d21: No bootstrap required, opened a new log
I20260926 01:03:55.650983 25706 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47b52894892d4626b78c60d6675d4d21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47b52894892d4626b78c60d6675d4d21" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 44911 } } peers { permanent_uuid: "ddaed7495f1a476ca21d0a231b8ea7ea" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 39535 } } peers { permanent_uuid: "0131d0575c9f4ae09eef7919b417e037" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 39741 } }
I20260926 01:03:55.651114 25706 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47b52894892d4626b78c60d6675d4d21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47b52894892d4626b78c60d6675d4d21, State: Initialized, Role: FOLLOWER
I20260926 01:03:55.651211 25706 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47b52894892d4626b78c60d6675d4d21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47b52894892d4626b78c60d6675d4d21" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 44911 } } peers { permanent_uuid: "ddaed7495f1a476ca21d0a231b8ea7ea" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 39535 } } peers { permanent_uuid: "0131d0575c9f4ae09eef7919b417e037" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 39741 } }
I20260926 01:03:55.651501 25877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47b52894892d4626b78c60d6675d4d21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47b52894892d4626b78c60d6675d4d21" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 44911 } } peers { permanent_uuid: "ddaed7495f1a476ca21d0a231b8ea7ea" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 39535 } } peers { permanent_uuid: "0131d0575c9f4ae09eef7919b417e037" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 39741 } } }
I20260926 01:03:55.651571 25706 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47b52894892d4626b78c60d6675d4d21 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:03:55.651629 25877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47b52894892d4626b78c60d6675d4d21 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:03:55.651656 25680 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47b52894892d4626b78c60d6675d4d21 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 01:03:55.651818 25680 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47b52894892d4626b78c60d6675d4d21 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:03:55.652626 25680 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47b52894892d4626b78c60d6675d4d21 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47b52894892d4626b78c60d6675d4d21" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 44911 } } peers { permanent_uuid: "ddaed7495f1a476ca21d0a231b8ea7ea" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 39535 } } peers { permanent_uuid: "0131d0575c9f4ae09eef7919b417e037" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 39741 } }
W20260926 01:03:55.653647 25888 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 47b52894892d4626b78c60d6675d4d21: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:03:55.653753 25888 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:03:55.654234 25831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "47b52894892d4626b78c60d6675d4d21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0131d0575c9f4ae09eef7919b417e037"
I20260926 01:03:55.654362 25831 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0131d0575c9f4ae09eef7919b417e037 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:03:55.655463 25831 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0131d0575c9f4ae09eef7919b417e037 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47b52894892d4626b78c60d6675d4d21 in term 1.
I20260926 01:03:55.654412 25637 master_runner.cc:186] Error getting master registration for 127.3.75.61:39535: OK, instance_id { permanent_uuid: "ddaed7495f1a476ca21d0a231b8ea7ea" instance_seqno: 1790384635451984 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 01:03:55.656771 25637 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 01:03:55.656903 25680 leader_election.cc:288] T 00000000000000000000000000000000 P 47b52894892d4626b78c60d6675d4d21 [CANDIDATE]: Term 1 election: Requested vote from peers ddaed7495f1a476ca21d0a231b8ea7ea (127.3.75.61:39535), 0131d0575c9f4ae09eef7919b417e037 (127.3.75.60:39741)
I20260926 01:03:55.656955 25653 leader_election.cc:302] T 00000000000000000000000000000000 P 47b52894892d4626b78c60d6675d4d21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0131d0575c9f4ae09eef7919b417e037, 47b52894892d4626b78c60d6675d4d21; no voters:
I20260926 01:03:55.657086 25877 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47b52894892d4626b78c60d6675d4d21 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:03:55.657225 25877 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47b52894892d4626b78c60d6675d4d21 [term 1 LEADER]: Becoming Leader. State: Replica: 47b52894892d4626b78c60d6675d4d21, State: Running, Role: LEADER
I20260926 01:03:55.657301 25877 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47b52894892d4626b78c60d6675d4d21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47b52894892d4626b78c60d6675d4d21" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 44911 } } peers { permanent_uuid: "ddaed7495f1a476ca21d0a231b8ea7ea" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 39535 } } peers { permanent_uuid: "0131d0575c9f4ae09eef7919b417e037" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 39741 } }
I20260926 01:03:55.657711 25877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47b52894892d4626b78c60d6675d4d21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47b52894892d4626b78c60d6675d4d21. Latest consensus state: current_term: 1 leader_uuid: "47b52894892d4626b78c60d6675d4d21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47b52894892d4626b78c60d6675d4d21" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 44911 } } peers { permanent_uuid: "ddaed7495f1a476ca21d0a231b8ea7ea" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 39535 } } peers { permanent_uuid: "0131d0575c9f4ae09eef7919b417e037" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 39741 } } }
I20260926 01:03:55.657799 25877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47b52894892d4626b78c60d6675d4d21 [sys.catalog]: This master's current role is: LEADER
I20260926 01:03:55.663508 25896 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:03:55.663739 25896 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:03:55.665871 25831 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0131d0575c9f4ae09eef7919b417e037 [term 1 FOLLOWER]: Refusing update from remote peer 47b52894892d4626b78c60d6675d4d21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:03:55.666384 25890 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 47b52894892d4626b78c60d6675d4d21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0131d0575c9f4ae09eef7919b417e037" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 39741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:03:55.667754 25710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:55.658418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43314 (local address 127.3.75.62:44911)
0926 01:03:55.658454 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:55.658456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:55.658942 (+ 486us) server_negotiation.cc:407] Connection header received
0926 01:03:55.659040 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:55.659041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:55.659050 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:55.659078 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:55.659767 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:55.660522 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:55.661584 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:55.661879 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:55.661921 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:55.663610 (+ 1689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:55.663611 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:55.663612 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:03:55.663618 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:55.665344 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:55.666763 (+ 1419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:55.666765 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:55.666765 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:55.666827 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:55.667030 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:55.667031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:55.667031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:55.667154 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:55.667240 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:55.667581 (+ 341us) server_negotiation.cc:299] Negotiation successful
0926 01:03:55.667652 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 01:03:55.667781 25863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0131d0575c9f4ae09eef7919b417e037 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47b52894892d4626b78c60d6675d4d21. Latest consensus state: current_term: 1 leader_uuid: "47b52894892d4626b78c60d6675d4d21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47b52894892d4626b78c60d6675d4d21" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 44911 } } peers { permanent_uuid: "ddaed7495f1a476ca21d0a231b8ea7ea" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 39535 } } peers { permanent_uuid: "0131d0575c9f4ae09eef7919b417e037" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 39741 } } }
I20260926 01:03:55.667865 25863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0131d0575c9f4ae09eef7919b417e037 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:03:55.672711 25877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47b52894892d4626b78c60d6675d4d21 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "47b52894892d4626b78c60d6675d4d21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47b52894892d4626b78c60d6675d4d21" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 44911 } } peers { permanent_uuid: "ddaed7495f1a476ca21d0a231b8ea7ea" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 39535 } } peers { permanent_uuid: "0131d0575c9f4ae09eef7919b417e037" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 39741 } } }
I20260926 01:03:55.672792 25877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47b52894892d4626b78c60d6675d4d21 [sys.catalog]: This master's current role is: LEADER
I20260926 01:03:55.673295 25863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0131d0575c9f4ae09eef7919b417e037 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "47b52894892d4626b78c60d6675d4d21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47b52894892d4626b78c60d6675d4d21" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 44911 } } peers { permanent_uuid: "ddaed7495f1a476ca21d0a231b8ea7ea" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 39535 } } peers { permanent_uuid: "0131d0575c9f4ae09eef7919b417e037" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 39741 } } }
I20260926 01:03:55.673389 25863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0131d0575c9f4ae09eef7919b417e037 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:03:55.673640 25901 mvcc.cc:205] Tried to move back new op lower bound from 7333415467684024320 to 7333415467652972544. Current Snapshot: MvccSnapshot[applied={T|T < 7333415467684024320}]
I20260926 01:03:55.673918 25900 mvcc.cc:205] Tried to move back new op lower bound from 7333415467684024320 to 7333415467652972544. Current Snapshot: MvccSnapshot[applied={T|T < 7333415467684024320}]
I20260926 01:03:55.675307 25896 catalog_manager.cc:1347] Generated new cluster ID: 2949b1a5fa6b45d3a1fdd45b27fcc5fc
I20260926 01:03:55.675356 25896 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:03:55.675688 25861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:55.658694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40292 (local address 127.3.75.60:39741)
0926 01:03:55.658731 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:55.658733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:55.668383 (+ 9650us) server_negotiation.cc:407] Connection header received
0926 01:03:55.668467 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:55.668468 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:55.668479 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:55.668508 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:55.669080 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:55.669823 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:55.670811 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:55.671119 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:55.671156 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:55.672512 (+ 1356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:55.672513 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:55.672513 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:03:55.672519 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:55.674308 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:55.674760 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:55.674761 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:55.674762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:55.674815 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:55.675045 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:55.675046 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:55.675046 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:55.675170 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:55.675236 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:55.675509 (+ 273us) server_negotiation.cc:299] Negotiation successful
0926 01:03:55.675585 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 01:03:55.679841 25786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:55.658066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51374 (local address 127.3.75.61:39535)
0926 01:03:55.658109 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:55.658111 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:55.658438 (+ 327us) server_negotiation.cc:407] Connection header received
0926 01:03:55.658598 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:55.658599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:55.658621 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:55.658653 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:55.659989 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:55.660725 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:55.662340 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:55.662652 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:55.662707 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:55.676842 (+ 14135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:55.676844 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:55.676844 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:03:55.676852 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:55.678489 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:55.678905 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:55.678906 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:55.678906 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:55.678961 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:55.679162 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:55.679162 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:55.679163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:55.679323 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:55.679404 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:55.679651 (+ 247us) server_negotiation.cc:299] Negotiation successful
0926 01:03:55.679743 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 01:03:55.686540 25781 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.75.60" port: 39741 } attempt: 3
I20260926 01:03:55.688268 25896 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:03:55.689059 25896 catalog_manager.cc:1504] Loading token signing keys...
Sep 26 01:03:55 dist-test-slave-7v10 krb5kdc[25624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384635, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.61@KRBTEST.COM for kudu/127.3.75.60@KRBTEST.COM
I20260926 01:03:55.693516 25896 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47b52894892d4626b78c60d6675d4d21: Generated new TSK 0
I20260926 01:03:55.693755 25896 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:03:55.695226 25784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:55.686809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.60:39741 (local address 127.0.0.1:40308)
0926 01:03:55.686871 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:55.686882 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:55.686931 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:55.687192 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:55.687193 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:55.687363 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:03:55.687409 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:55.687411 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:55.688309 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:55.688310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:55.689114 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:55.689117 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:55.689687 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:55.689689 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:55.689767 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:55.690283 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:03:55.690288 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:03:55.691776 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:03:55.693855 (+ 2079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:03:55.693857 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:03:55.693857 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:03:55.694076 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:03:55.694360 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:03:55.694361 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:03:55.694361 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:03:55.694417 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:03:55.694823 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:03:55.694824 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:03:55.694955 (+ 131us) client_negotiation.cc:769] Sending connection context
0926 01:03:55.695027 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 01:03:55.695124 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 01:03:55.695710 25861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:55.687028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40308 (local address 127.3.75.60:39741)
0926 01:03:55.687078 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:55.687080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:55.687088 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:03:55.687118 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:55.687119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:55.687129 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:55.687182 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:55.687502 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:55.688226 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:55.689227 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:55.689547 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:55.689609 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:55.691884 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:55.691885 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:55.691885 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:03:55.691892 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:55.693744 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:55.694166 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:55.694167 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:55.694168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:55.694259 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:55.694491 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:55.694492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:55.694493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:55.694698 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:55.695326 (+ 628us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:55.695415 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 01:03:55.695543 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 01:03:55.697201 25781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddaed7495f1a476ca21d0a231b8ea7ea: Bootstrap starting.
I20260926 01:03:55.697935 25781 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ddaed7495f1a476ca21d0a231b8ea7ea: Neither blocks nor log segments found. Creating new log.
I20260926 01:03:55.698228 25781 log.cc:824] T 00000000000000000000000000000000 P ddaed7495f1a476ca21d0a231b8ea7ea: Log is configured to *not* fsync() on all Append() calls
I20260926 01:03:55.699162 25781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddaed7495f1a476ca21d0a231b8ea7ea: No bootstrap required, opened a new log
I20260926 01:03:55.701128 25781 raft_consensus.cc:348] T 00000000000000000000000000000000 P ddaed7495f1a476ca21d0a231b8ea7ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47b52894892d4626b78c60d6675d4d21" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 44911 } } peers { permanent_uuid: "ddaed7495f1a476ca21d0a231b8ea7ea" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 39535 } } peers { permanent_uuid: "0131d0575c9f4ae09eef7919b417e037" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 39741 } }
I20260926 01:03:55.701279 25781 raft_consensus.cc:729] T 00000000000000000000000000000000 P ddaed7495f1a476ca21d0a231b8ea7ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddaed7495f1a476ca21d0a231b8ea7ea, State: Initialized, Role: FOLLOWER
I20260926 01:03:55.701364 25781 consensus_queue.cc:260] T 00000000000000000000000000000000 P ddaed7495f1a476ca21d0a231b8ea7ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47b52894892d4626b78c60d6675d4d21" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 44911 } } peers { permanent_uuid: "ddaed7495f1a476ca21d0a231b8ea7ea" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 39535 } } peers { permanent_uuid: "0131d0575c9f4ae09eef7919b417e037" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 39741 } }
I20260926 01:03:55.701683 25781 sys_catalog.cc:564] T 00000000000000000000000000000000 P ddaed7495f1a476ca21d0a231b8ea7ea [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:03:55.701715 25755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "47b52894892d4626b78c60d6675d4d21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ddaed7495f1a476ca21d0a231b8ea7ea"
I20260926 01:03:55.701756 25754 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ddaed7495f1a476ca21d0a231b8ea7ea [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:03:55.702847 25754 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ddaed7495f1a476ca21d0a231b8ea7ea [term 1 FOLLOWER]: Refusing update from remote peer 47b52894892d4626b78c60d6675d4d21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:03:55.703919 25755 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ddaed7495f1a476ca21d0a231b8ea7ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47b52894892d4626b78c60d6675d4d21 in term 1.
I20260926 01:03:55.704432 25904 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 47b52894892d4626b78c60d6675d4d21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddaed7495f1a476ca21d0a231b8ea7ea" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 39535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:03:55.701665 25907 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddaed7495f1a476ca21d0a231b8ea7ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47b52894892d4626b78c60d6675d4d21" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 44911 } } peers { permanent_uuid: "ddaed7495f1a476ca21d0a231b8ea7ea" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 39535 } } peers { permanent_uuid: "0131d0575c9f4ae09eef7919b417e037" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 39741 } } }
I20260926 01:03:55.704723 25907 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddaed7495f1a476ca21d0a231b8ea7ea [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:03:55.709199 25921 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddaed7495f1a476ca21d0a231b8ea7ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47b52894892d4626b78c60d6675d4d21. Latest consensus state: current_term: 1 leader_uuid: "47b52894892d4626b78c60d6675d4d21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47b52894892d4626b78c60d6675d4d21" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 44911 } } peers { permanent_uuid: "ddaed7495f1a476ca21d0a231b8ea7ea" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 39535 } } peers { permanent_uuid: "0131d0575c9f4ae09eef7919b417e037" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 39741 } } }
I20260926 01:03:55.709290 25921 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddaed7495f1a476ca21d0a231b8ea7ea [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:03:55.709326 25921 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddaed7495f1a476ca21d0a231b8ea7ea [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "47b52894892d4626b78c60d6675d4d21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47b52894892d4626b78c60d6675d4d21" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 44911 } } peers { permanent_uuid: "ddaed7495f1a476ca21d0a231b8ea7ea" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 39535 } } peers { permanent_uuid: "0131d0575c9f4ae09eef7919b417e037" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 39741 } } }
I20260926 01:03:55.709368 25921 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddaed7495f1a476ca21d0a231b8ea7ea [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:03:55.711685 25918 catalog_manager.cc:1259] Loaded cluster ID: 2949b1a5fa6b45d3a1fdd45b27fcc5fc
I20260926 01:03:55.711776 25918 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ddaed7495f1a476ca21d0a231b8ea7ea: loading cluster ID for follower catalog manager: success
I20260926 01:03:55.713013 25918 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ddaed7495f1a476ca21d0a231b8ea7ea: acquiring CA information for follower catalog manager: success
I20260926 01:03:55.714561 25918 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ddaed7495f1a476ca21d0a231b8ea7ea: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 01:03:55.740952 25786 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:03:55.739509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51378 (local address 127.3.75.61:39535)
0926 01:03:55.739548 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:55.739550 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:55.740227 (+ 677us) server_negotiation.cc:407] Connection header received
0926 01:03:55.740335 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:55.740336 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:55.740347 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:55.740378 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:55.740869 (+ 491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51378: BlockingRecv error: recv got EOF from 127.0.0.1:51378 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 01:03:55.741709 25710 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:03:55.739233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43330 (local address 127.3.75.62:44911)
0926 01:03:55.739299 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:55.739301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:55.741197 (+ 1896us) server_negotiation.cc:407] Connection header received
0926 01:03:55.741301 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:55.741302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:55.741312 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:55.741342 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:55.741649 (+ 307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43330: BlockingRecv error: recv got EOF from 127.0.0.1:43330 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260926 01:03:55.742411 25861 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:03:55.739968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40316 (local address 127.3.75.60:39741)
0926 01:03:55.740009 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:55.740010 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:55.741921 (+ 1911us) server_negotiation.cc:407] Connection header received
0926 01:03:55.742016 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:55.742017 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:55.742027 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:55.742056 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:55.742355 (+ 299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40316: BlockingRecv error: recv got EOF from 127.0.0.1:40316 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 01:03:55.747649 25786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:55.744061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51382 (local address 127.3.75.61:39535)
0926 01:03:55.744103 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:55.744104 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:55.744314 (+ 210us) server_negotiation.cc:407] Connection header received
0926 01:03:55.744418 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:55.744419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:55.744458 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:55.744500 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:03:55.744689 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:55.745470 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:55.746644 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:55.746987 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:55.747030 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:55.747266 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:03:55.747399 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:03:55.747447 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:55.747558 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 01:03:55.747577 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
W20260926 01:03:55.750787 25861 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:03:55.750047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40330 (local address 127.3.75.60:39741)
0926 01:03:55.750084 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:55.750086 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:55.750260 (+ 174us) server_negotiation.cc:407] Connection header received
0926 01:03:55.750363 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:55.750364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:55.750383 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:55.750413 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:55.750734 (+ 321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40330: BlockingRecv error: recv got EOF from 127.0.0.1:40330 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 01:03:55.756433 25786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:55.753018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51394 (local address 127.3.75.61:39535)
0926 01:03:55.753050 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:55.753052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:55.753340 (+ 288us) server_negotiation.cc:407] Connection header received
0926 01:03:55.753442 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:55.753442 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:55.753453 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:55.753479 (+ 26us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:03:55.753629 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:55.754352 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:55.755628 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:55.755927 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:55.755968 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:55.756159 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:03:55.756207 (+ 48us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:03:55.756235 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:55.756339 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 01:03:55.756358 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260926 01:03:55.759570 25861 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:03:55.758764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40338 (local address 127.3.75.60:39741)
0926 01:03:55.758805 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:55.758807 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:55.759008 (+ 201us) server_negotiation.cc:407] Connection header received
0926 01:03:55.759104 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:55.759105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:55.759117 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:55.759147 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:55.759517 (+ 370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40338: BlockingRecv error: recv got EOF from 127.0.0.1:40338 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 01:03:55.765929 25786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:55.762694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51400 (local address 127.3.75.61:39535)
0926 01:03:55.762733 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:55.762736 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:55.762986 (+ 250us) server_negotiation.cc:407] Connection header received
0926 01:03:55.763081 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:55.763082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:55.763092 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:55.763120 (+ 28us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:03:55.763297 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:55.764059 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:55.765158 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:55.765459 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:55.765497 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:55.765686 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:03:55.765722 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:03:55.765749 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:55.765844 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 01:03:55.765861 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260926 01:03:55.768901 25861 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:03:55.768215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40352 (local address 127.3.75.60:39741)
0926 01:03:55.768249 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:55.768251 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:55.768407 (+ 156us) server_negotiation.cc:407] Connection header received
0926 01:03:55.768495 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:55.768496 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:55.768506 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:55.768534 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:55.768849 (+ 315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40352: BlockingRecv error: recv got EOF from 127.0.0.1:40352 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 01:03:55.777364 25786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:55.773984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51412 (local address 127.3.75.61:39535)
0926 01:03:55.774023 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:55.774026 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:55.774281 (+ 255us) server_negotiation.cc:407] Connection header received
0926 01:03:55.774380 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:55.774381 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:55.774391 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:55.774418 (+ 27us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:03:55.774596 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:55.775426 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:55.776565 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:55.776848 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:55.776889 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:55.777080 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:03:55.777113 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:03:55.777140 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:55.777260 (+ 120us) server_negotiation.cc:299] Negotiation successful
0926 01:03:55.777283 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 01:03:55.780581 25861 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:03:55.779779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40366 (local address 127.3.75.60:39741)
0926 01:03:55.779822 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:55.779823 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:55.779992 (+ 169us) server_negotiation.cc:407] Connection header received
0926 01:03:55.780125 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:55.780126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:55.780138 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:55.780168 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:55.780524 (+ 356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40366: BlockingRecv error: recv got EOF from 127.0.0.1:40366 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 01:03:55.792919 25786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:55.789768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51426 (local address 127.3.75.61:39535)
0926 01:03:55.789806 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:55.789808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:55.790046 (+ 238us) server_negotiation.cc:407] Connection header received
0926 01:03:55.790141 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:55.790142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:55.790152 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:55.790192 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:03:55.790360 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:55.791116 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:55.792167 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:55.792455 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:55.792492 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:55.792670 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:03:55.792702 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:03:55.792728 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:55.792828 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 01:03:55.792846 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260926 01:03:55.795933 25861 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:03:55.795161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40370 (local address 127.3.75.60:39741)
0926 01:03:55.795212 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:55.795214 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:55.795395 (+ 181us) server_negotiation.cc:407] Connection header received
0926 01:03:55.795491 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:55.795492 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:55.795504 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:55.795533 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:55.795875 (+ 342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40370: BlockingRecv error: recv got EOF from 127.0.0.1:40370 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 01:03:55.816790 25786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:55.813344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51440 (local address 127.3.75.61:39535)
0926 01:03:55.813391 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:55.813393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:55.813714 (+ 321us) server_negotiation.cc:407] Connection header received
0926 01:03:55.813827 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:55.813828 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:55.813839 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:55.813869 (+ 30us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:03:55.814069 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:55.814825 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:55.815957 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:55.816254 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:55.816301 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:55.816526 (+ 225us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:03:55.816559 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:03:55.816587 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:55.816692 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 01:03:55.816712 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 01:03:55.820137 25861 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:03:55.819344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40380 (local address 127.3.75.60:39741)
0926 01:03:55.819387 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:55.819389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:55.819576 (+ 187us) server_negotiation.cc:407] Connection header received
0926 01:03:55.819673 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:55.819674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:55.819687 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:55.819718 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:55.820080 (+ 362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40380: BlockingRecv error: recv got EOF from 127.0.0.1:40380 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 01:03:55.856339 25786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:55.853586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51454 (local address 127.3.75.61:39535)
0926 01:03:55.853654 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:55.853656 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:55.853775 (+ 119us) server_negotiation.cc:407] Connection header received
0926 01:03:55.853845 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:55.853846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:55.853856 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:55.853920 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:03:55.854054 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:55.854505 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:55.855657 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:55.855837 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:55.855893 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:55.856083 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:03:55.856113 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:03:55.856148 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:55.856240 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 01:03:55.856255 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260926 01:03:55.858817 25861 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:03:55.858149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40390 (local address 127.3.75.60:39741)
0926 01:03:55.858201 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:55.858203 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:55.858256 (+ 53us) server_negotiation.cc:407] Connection header received
0926 01:03:55.858324 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:55.858325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:55.858334 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:55.858377 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:55.858732 (+ 355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40390: BlockingRecv error: recv got EOF from 127.0.0.1:40390 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 01:03:55.926218 25786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:55.923897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51460 (local address 127.3.75.61:39535)
0926 01:03:55.923974 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:55.923976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:55.924044 (+ 68us) server_negotiation.cc:407] Connection header received
0926 01:03:55.924095 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:55.924096 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:55.924105 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:55.924167 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:03:55.924279 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:55.924794 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:55.925617 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:55.925812 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:55.925864 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:55.925978 (+ 114us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:03:55.926004 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:03:55.926041 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:55.926088 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 01:03:55.926103 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260926 01:03:55.928427 25861 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:03:55.927860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40406 (local address 127.3.75.60:39741)
0926 01:03:55.927930 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:55.927932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:55.927979 (+ 47us) server_negotiation.cc:407] Connection header received
0926 01:03:55.928024 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:55.928024 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:55.928035 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:55.928080 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:55.928352 (+ 272us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40406: BlockingRecv error: recv got EOF from 127.0.0.1:40406 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 01:03:56.059741 25786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:56.057537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51470 (local address 127.3.75.61:39535)
0926 01:03:56.057628 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:56.057630 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:56.057688 (+ 58us) server_negotiation.cc:407] Connection header received
0926 01:03:56.057765 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:56.057766 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:56.057774 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:56.057815 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:03:56.057940 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:56.058406 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:56.059155 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:56.059362 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:56.059411 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:56.059545 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:03:56.059571 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:03:56.059609 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:56.059665 (+ 56us) server_negotiation.cc:299] Negotiation successful
0926 01:03:56.059681 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260926 01:03:56.062077 25861 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:03:56.061463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40410 (local address 127.3.75.60:39741)
0926 01:03:56.061525 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:56.061527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:56.061578 (+ 51us) server_negotiation.cc:407] Connection header received
0926 01:03:56.061654 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:56.061655 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:56.061666 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:56.061722 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:56.062022 (+ 300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40410: BlockingRecv error: recv got EOF from 127.0.0.1:40410 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 01:03:56.286911 25904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47b52894892d4626b78c60d6675d4d21 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "47b52894892d4626b78c60d6675d4d21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47b52894892d4626b78c60d6675d4d21" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 44911 } } peers { permanent_uuid: "ddaed7495f1a476ca21d0a231b8ea7ea" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 39535 } } peers { permanent_uuid: "0131d0575c9f4ae09eef7919b417e037" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 39741 } } }
I20260926 01:03:56.287083 25904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47b52894892d4626b78c60d6675d4d21 [sys.catalog]: This master's current role is: LEADER
I20260926 01:03:56.321585 25786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:56.319430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51484 (local address 127.3.75.61:39535)
0926 01:03:56.319502 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:56.319503 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:56.319531 (+ 28us) server_negotiation.cc:407] Connection header received
0926 01:03:56.319611 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:56.319612 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:56.319620 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:56.319665 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:03:56.319803 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:56.320274 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:56.320980 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:56.321145 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:56.321243 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:56.321381 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:03:56.321406 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:03:56.321443 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:56.321513 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 01:03:56.321527 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260926 01:03:56.323825 25861 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:03:56.323341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40418 (local address 127.3.75.60:39741)
0926 01:03:56.323400 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:56.323402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:56.323411 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:03:56.323435 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:56.323436 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:56.323446 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:56.323509 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:56.323760 (+ 251us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40418: BlockingRecv error: recv got EOF from 127.0.0.1:40418 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 01:03:56.634506 25874 catalog_manager.cc:1259] Loaded cluster ID: 2949b1a5fa6b45d3a1fdd45b27fcc5fc
I20260926 01:03:56.634578 25874 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0131d0575c9f4ae09eef7919b417e037: loading cluster ID for follower catalog manager: success
I20260926 01:03:56.635396 25874 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0131d0575c9f4ae09eef7919b417e037: acquiring CA information for follower catalog manager: success
I20260926 01:03:56.635710 25874 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0131d0575c9f4ae09eef7919b417e037: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 01:03:56.839178 26044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:56.836948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51498 (local address 127.3.75.61:39535)
0926 01:03:56.837098 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:56.837100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:56.837110 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:03:56.837173 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:56.837174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:56.837185 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:56.837243 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:03:56.837363 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:56.837840 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:56.838547 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:56.838736 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:56.838787 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:56.838966 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:03:56.838993 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:03:56.839051 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:56.839097 (+ 46us) server_negotiation.cc:299] Negotiation successful
0926 01:03:56.839113 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":79,"threads_started":1}
I20260926 01:03:56.843242 26050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:56.840966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40432 (local address 127.3.75.60:39741)
0926 01:03:56.841071 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:56.841073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:56.841096 (+ 23us) server_negotiation.cc:407] Connection header received
0926 01:03:56.841131 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:56.841132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:56.841176 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:56.841235 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:03:56.841318 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:56.841774 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:56.842406 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:56.842603 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:56.842654 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:56.842833 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:03:56.842992 (+ 159us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:03:56.843054 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:56.843104 (+ 50us) server_negotiation.cc:299] Negotiation successful
0926 01:03:56.843126 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":63,"threads_started":1}
I20260926 01:03:56.847767 26044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:56.844686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51512 (local address 127.3.75.61:39535)
0926 01:03:56.844725 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:56.844727 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:56.845049 (+ 322us) server_negotiation.cc:407] Connection header received
0926 01:03:56.845124 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:56.845125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:56.845135 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:56.845187 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:03:56.845310 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:56.845849 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:56.846945 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:56.847253 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:56.847290 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:56.847480 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:03:56.847511 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:03:56.847539 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:56.847670 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 01:03:56.847693 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 01:03:56.847787 26050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:56.844886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40446 (local address 127.3.75.60:39741)
0926 01:03:56.844925 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:56.844927 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:56.845324 (+ 397us) server_negotiation.cc:407] Connection header received
0926 01:03:56.845347 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:56.845348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:56.845357 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:56.845405 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:03:56.845533 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:56.846197 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:56.846998 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:56.847235 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:56.847311 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:56.847466 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:03:56.847499 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:03:56.847564 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:56.847701 (+ 137us) server_negotiation.cc:299] Negotiation successful
0926 01:03:56.847722 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 01:03:56.848359 26056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:56.844674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43336 (local address 127.3.75.62:44911)
0926 01:03:56.844779 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:56.844781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:56.844923 (+ 142us) server_negotiation.cc:407] Connection header received
0926 01:03:56.844994 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:56.844995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:56.845030 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:56.845127 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:03:56.845260 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:56.846035 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:56.847037 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:56.847391 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:56.847432 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:56.847677 (+ 245us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:03:56.847783 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:03:56.847852 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:56.848232 (+ 380us) server_negotiation.cc:299] Negotiation successful
0926 01:03:56.848261 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":56,"threads_started":1}
I20260926 01:03:56.849890 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 25637
I20260926 01:03:56.855207 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 25713
I20260926 01:03:56.860450 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 25788
2026-09-26T01:03:56Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2846 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:03:56 dist-test-slave-7v10 krb5kdc[26061](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:03:56 dist-test-slave-7v10 krb5kdc[26061](info): set up 2 sockets
Sep 26 01:03:56 dist-test-slave-7v10 krb5kdc[26061](info): commencing operation
krb5kdc: starting...
W20260926 01:03:57.978421 3372 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/krb5kdc/test-user.keytab.
Sep 26 01:03:58 dist-test-slave-7v10 krb5kdc[26061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:03:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:03:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.75.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab.
Entry for principal kudu/127.3.75.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab.
I20260926 01:03:58.094240 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:36497
--webserver_interface=127.3.75.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:35693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.75.62:36497,127.3.75.61:45861,127.3.75.60:35451
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:03:58.174147 26074 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 01:03:58.174377 26074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:03:58.174417 26074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:03:58.174474 26074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:03:58.175580 26074 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:03:58.175652 26074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:03:58.175684 26074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:03:58.175717 26074 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:03:58.175748 26074 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:03:58.177214 26074 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:35693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.75.62:36497,127.3.75.61:45861,127.3.75.60:35451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.62.keytab
--principal=kudu/127.3.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.62:36497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:03:58.177500 26074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:03:58.177716 26074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 01:03:58.182430 26074 server_base.cc:1034] running on GCE node
W20260926 01:03:58.182386 26079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:58.182396 26080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:58.182374 26082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:03:58.182936 26074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:03:58.183223 26074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:03:58.184381 26074 hybrid_clock.cc:648] HybridClock initialized: now 1790384638184368 us; error 43 us; skew 500 ppm
Sep 26 01:03:58 dist-test-slave-7v10 krb5kdc[26061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384638, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:03:58.186800 26074 init.cc:377] Logged in from keytab as kudu/127.3.75.62@KRBTEST.COM (short username kudu)
I20260926 01:03:58.187263 26074 webserver.cc:460] Webserver started at http://127.3.75.62:41645/ using document root <none> and password file <none>
I20260926 01:03:58.187486 26074 fs_manager.cc:359] Metadata directory not provided
I20260926 01:03:58.187528 26074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:03:58.187693 26074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:03:58.188928 26074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/minicluster-data/master-0/data/instance:
uuid: "b2fdca5f39f54b378c7340247e37b35d"
format_stamp: "Formatted at 2026-09-26 01:03:58 on dist-test-slave-7v10"
server_key: "7814b6c1b7698507e359ec6e42404fdf"
server_key_iv: "41a3e02c08266efd113d6df0b687d251"
server_key_version: "encryptionkey@0"
I20260926 01:03:58.189309 26074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/minicluster-data/master-0/wal/instance:
uuid: "b2fdca5f39f54b378c7340247e37b35d"
format_stamp: "Formatted at 2026-09-26 01:03:58 on dist-test-slave-7v10"
server_key: "7814b6c1b7698507e359ec6e42404fdf"
server_key_iv: "41a3e02c08266efd113d6df0b687d251"
server_key_version: "encryptionkey@0"
I20260926 01:03:58.190613 26074 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 01:03:58.191383 26089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:58.191582 26074 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:58.191661 26074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/minicluster-data/master-0/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/minicluster-data/master-0/wal
uuid: "b2fdca5f39f54b378c7340247e37b35d"
format_stamp: "Formatted at 2026-09-26 01:03:58 on dist-test-slave-7v10"
server_key: "7814b6c1b7698507e359ec6e42404fdf"
server_key_iv: "41a3e02c08266efd113d6df0b687d251"
server_key_version: "encryptionkey@0"
I20260926 01:03:58.191713 26074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:03:58.206377 26074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:03:58.208814 26074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:03:58.208935 26074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:03:58.212941 26074 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.62:36497
I20260926 01:03:58.213047 26143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.62:36497 every 8 connection(s)
I20260926 01:03:58.213362 26074 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/minicluster-data/master-0/data/info.pb
I20260926 01:03:58.214466 26144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:03:58.216195 26144 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.75.62" port: 36497 } has no permanent_uuid. Determining permanent_uuid...
I20260926 01:03:58.219107 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 26074
I20260926 01:03:58.219193 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/minicluster-data/master-0/wal/instance
I20260926 01:03:58.219409 3372 external_mini_cluster.cc:1428] Setting key 523e9ceb9d43af2dc973c644686a65f5
Sep 26 01:03:58 dist-test-slave-7v10 krb5kdc[26061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384638, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.62@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 01:03:58.226994 26146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:58.216908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:36497 (local address 127.0.0.1:43254)
0926 01:03:58.217113 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:58.217153 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:58.217627 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:58.217953 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:58.217955 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:58.218236 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:03:58.218404 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:58.218409 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:58.219747 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:58.219748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:58.220516 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:58.220520 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:58.221068 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:58.221069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:58.221155 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:58.221694 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:03:58.221702 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:03:58.223229 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:03:58.225409 (+ 2180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:03:58.225415 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:03:58.225417 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:03:58.225607 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:03:58.225895 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:03:58.225896 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:03:58.225896 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:03:58.225974 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:03:58.226439 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:03:58.226445 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:03:58.226506 (+ 61us) client_negotiation.cc:769] Sending connection context
0926 01:03:58.226621 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 01:03:58.226758 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":83,"threads_started":1}
I20260926 01:03:58.227011 26147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:58.217090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43254 (local address 127.3.75.62:36497)
0926 01:03:58.217213 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:58.217216 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:58.217243 (+ 27us) server_negotiation.cc:407] Connection header received
0926 01:03:58.217809 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:58.217811 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:58.217829 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:58.217915 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:58.218513 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:58.219638 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:58.220630 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:58.220949 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:58.221039 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:58.223323 (+ 2284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:58.223332 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:58.223332 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:03:58.223340 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:58.225316 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:58.225700 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:58.225704 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:58.225704 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:58.225818 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:58.226038 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:58.226039 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:58.226039 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:58.226343 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:58.226439 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:58.226690 (+ 251us) server_negotiation.cc:299] Negotiation successful
0926 01:03:58.226841 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":87,"thread_start_us":61,"threads_started":1}
WARNING: no policy specified for kudu/127.3.75.61@KRBTEST.COM; defaulting to no policy
I20260926 01:03:58.227890 26144 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.75.61" port: 45861 } has no permanent_uuid. Determining permanent_uuid...
I20260926 01:03:58.228356 26146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:58.228242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.61:45861 (local address 127.0.0.1:34692)
0926 01:03:58.228298 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:58.228338 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.61:45861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 01:03:58.228559 26093 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.75.61:45861: connect: Connection refused (error 111)
I20260926 01:03:58.228965 26146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:58.228882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.61:45861 (local address 127.0.0.1:34696)
0926 01:03:58.228924 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:58.228949 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.61:45861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 01:03:58.229141 26144 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.75.61:45861: Network error: Client connection negotiation failed: client connection to 127.3.75.61:45861: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.61.keytab.
Entry for principal kudu/127.3.75.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.61.keytab.
I20260926 01:03:58.243871 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.61:45861
--webserver_interface=127.3.75.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.61.keytab
--principal=kudu/127.3.75.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:35693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.75.62:36497,127.3.75.61:45861,127.3.75.60:35451
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 01:03:58.278374 26144 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.75.61" port: 45861 } attempt: 1
I20260926 01:03:58.278932 26146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:58.278744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.61:45861 (local address 127.0.0.1:34704)
0926 01:03:58.278820 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:58.278855 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.61:45861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 01:03:58.279296 26146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:58.279184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.61:45861 (local address 127.0.0.1:34706)
0926 01:03:58.279231 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:58.279266 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.61:45861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 01:03:58.279378 26144 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.75.61:45861: Network error: Client connection negotiation failed: client connection to 127.3.75.61:45861: connect: Connection refused (error 111)
W20260926 01:03:58.325299 26151 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 01:03:58.325532 26151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:03:58.325551 26151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:03:58.325572 26151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:03:58.326797 26151 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:03:58.326846 26151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:03:58.326860 26151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:03:58.326902 26151 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:03:58.326915 26151 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:03:58.328473 26151 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:35693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.75.62:36497,127.3.75.61:45861,127.3.75.60:35451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.61.keytab
--principal=kudu/127.3.75.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.61:45861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.75.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:03:58.328751 26151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:03:58.328994 26151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:03:58.333914 26157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:58.333920 26156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:58.333941 26159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:03:58.334151 26151 server_base.cc:1034] running on GCE node
I20260926 01:03:58.334379 26151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:03:58.334673 26151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:03:58.335836 26151 hybrid_clock.cc:648] HybridClock initialized: now 1790384638335831 us; error 34 us; skew 500 ppm
Sep 26 01:03:58 dist-test-slave-7v10 krb5kdc[26061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384638, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:03:58.338169 26151 init.cc:377] Logged in from keytab as kudu/127.3.75.61@KRBTEST.COM (short username kudu)
I20260926 01:03:58.338660 26151 webserver.cc:460] Webserver started at http://127.3.75.61:38203/ using document root <none> and password file <none>
I20260926 01:03:58.338917 26151 fs_manager.cc:359] Metadata directory not provided
I20260926 01:03:58.338994 26151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:03:58.339155 26151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:03:58.340240 26151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/minicluster-data/master-1/data/instance:
uuid: "d723d6446940468398b01e4dfa20068c"
format_stamp: "Formatted at 2026-09-26 01:03:58 on dist-test-slave-7v10"
server_key: "5090d549c1521d40c7d1f29c1c10c526"
server_key_iv: "f15b0178587c86a80de1bd9c58aa9963"
server_key_version: "encryptionkey@0"
I20260926 01:03:58.340613 26151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/minicluster-data/master-1/wal/instance:
uuid: "d723d6446940468398b01e4dfa20068c"
format_stamp: "Formatted at 2026-09-26 01:03:58 on dist-test-slave-7v10"
server_key: "5090d549c1521d40c7d1f29c1c10c526"
server_key_iv: "f15b0178587c86a80de1bd9c58aa9963"
server_key_version: "encryptionkey@0"
I20260926 01:03:58.341969 26151 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:03:58.342712 26166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:58.342909 26151 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:03:58.343020 26151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/minicluster-data/master-1/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/minicluster-data/master-1/wal
uuid: "d723d6446940468398b01e4dfa20068c"
format_stamp: "Formatted at 2026-09-26 01:03:58 on dist-test-slave-7v10"
server_key: "5090d549c1521d40c7d1f29c1c10c526"
server_key_iv: "f15b0178587c86a80de1bd9c58aa9963"
server_key_version: "encryptionkey@0"
I20260926 01:03:58.343104 26151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:03:58.347523 26144 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.75.61" port: 45861 } attempt: 2
I20260926 01:03:58.347957 26146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:58.347833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.61:45861 (local address 127.0.0.1:34708)
0926 01:03:58.347891 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:58.347929 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.61:45861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 01:03:58.348284 26146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:58.348198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.61:45861 (local address 127.0.0.1:34716)
0926 01:03:58.348241 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:58.348266 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.61:45861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 01:03:58.348405 26144 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.75.61:45861: Network error: Client connection negotiation failed: client connection to 127.3.75.61:45861: connect: Connection refused (error 111)
I20260926 01:03:58.365005 26151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:03:58.366748 26151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:03:58.366853 26151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:03:58.370546 26151 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.61:45861
I20260926 01:03:58.370607 26219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.61:45861 every 8 connection(s)
I20260926 01:03:58.371003 26151 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/minicluster-data/master-1/data/info.pb
I20260926 01:03:58.372022 26220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:03:58.373570 26220 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.75.62" port: 36497 } has no permanent_uuid. Determining permanent_uuid...
I20260926 01:03:58.378990 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 26151
I20260926 01:03:58.379098 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/minicluster-data/master-1/wal/instance
I20260926 01:03:58.379304 3372 external_mini_cluster.cc:1428] Setting key 7abaff63eb78376aedfbd8b6363aef0c
Sep 26 01:03:58 dist-test-slave-7v10 krb5kdc[26061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384638, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.61@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 01:03:58.382927 26147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:58.374267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43264 (local address 127.3.75.62:36497)
0926 01:03:58.374356 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:58.374358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:58.374471 (+ 113us) server_negotiation.cc:407] Connection header received
0926 01:03:58.374936 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:58.374937 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:58.374949 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:58.375000 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:58.375584 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:58.376077 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:58.377024 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:58.377224 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:58.377282 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:58.379805 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:58.379807 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:58.379807 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:03:58.379815 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:58.381523 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:58.381926 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:58.381927 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:58.381928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:58.381986 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:58.382253 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:58.382254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:58.382254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:58.382376 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:58.382452 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:58.382736 (+ 284us) server_negotiation.cc:299] Negotiation successful
0926 01:03:58.382817 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 01:03:58.383131 26222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:58.374211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:36497 (local address 127.0.0.1:43264)
0926 01:03:58.374354 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:58.374389 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:58.374818 (+ 429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:58.375051 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:58.375054 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:58.375293 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:03:58.375478 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:58.375482 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:58.376164 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:58.376165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:58.376944 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:58.376946 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:58.377313 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:58.377313 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:58.377423 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:58.377976 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:03:58.377985 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:03:58.379703 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:03:58.381609 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:03:58.381615 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:03:58.381618 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:03:58.381845 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:03:58.382063 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:03:58.382064 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:03:58.382064 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:03:58.382183 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:03:58.382444 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:03:58.382451 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:03:58.382591 (+ 140us) client_negotiation.cc:769] Sending connection context
0926 01:03:58.382704 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 01:03:58.382860 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":93,"threads_started":1}
I20260926 01:03:58.383723 26220 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.75.61" port: 45861 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.75.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.75.60@KRBTEST.COM" created.
Sep 26 01:03:58 dist-test-slave-7v10 krb5kdc[26061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384638, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.61@KRBTEST.COM for kudu/127.3.75.61@KRBTEST.COM
I20260926 01:03:58.394106 26224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:58.384057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34732 (local address 127.3.75.61:45861)
0926 01:03:58.384208 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:58.384211 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:58.384275 (+ 64us) server_negotiation.cc:407] Connection header received
0926 01:03:58.384342 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:58.384344 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:58.384354 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:58.384410 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:58.384755 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:58.385392 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:58.386375 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:58.386649 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:58.386712 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:58.390821 (+ 4109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:58.390831 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:58.390832 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:03:58.390839 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:58.392796 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:58.393217 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:58.393219 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:58.393220 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:58.393270 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:58.393468 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:58.393469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:58.393469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:58.393599 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:58.393703 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:58.393841 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 01:03:58.393983 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":77,"server-negotiator.queue_time_us":127,"thread_start_us":97,"threads_started":1}
I20260926 01:03:58.394121 26222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:58.383980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.61:45861 (local address 127.0.0.1:34732)
0926 01:03:58.384214 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:58.384225 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:58.384283 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:58.384441 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:58.384442 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:58.384609 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:03:58.384682 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:58.384684 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:58.385477 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:58.385478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:58.386298 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:58.386300 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:58.386725 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:58.386726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:58.386810 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:58.387184 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:03:58.387188 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:03:58.390703 (+ 3515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:03:58.392890 (+ 2187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:03:58.392891 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:03:58.392892 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:03:58.393125 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:03:58.393332 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:03:58.393333 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:03:58.393334 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:03:58.393407 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:03:58.393688 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:03:58.393689 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:03:58.393740 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 01:03:58.393817 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 01:03:58.393937 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217}
I20260926 01:03:58.394804 26220 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.75.60" port: 35451 } has no permanent_uuid. Determining permanent_uuid...
I20260926 01:03:58.395207 26222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:58.395066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.60:35451 (local address 127.0.0.1:58976)
0926 01:03:58.395119 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:58.395175 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.60:35451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 01:03:58.395356 26168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.75.60:35451: connect: Connection refused (error 111)
I20260926 01:03:58.395804 26222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:58.395682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.60:35451 (local address 127.0.0.1:58980)
0926 01:03:58.395741 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:58.395773 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.60:35451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 01:03:58.395943 26220 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.75.60:35451: Network error: Client connection negotiation failed: client connection to 127.3.75.60:35451: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.75.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.60.keytab.
Entry for principal kudu/127.3.75.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.60.keytab.
I20260926 01:03:58.406159 3372 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktuUsJs/build/release/bin/kudu
/tmp/dist-test-tasktuUsJs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.60:35451
--webserver_interface=127.3.75.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.60.keytab
--principal=kudu/127.3.75.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.75.20:35693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.75.62:36497,127.3.75.61:45861,127.3.75.60:35451
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 01:03:58.439567 26144 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.75.61" port: 45861 } attempt: 3
Sep 26 01:03:58 dist-test-slave-7v10 krb5kdc[26061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384638, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.62@KRBTEST.COM for kudu/127.3.75.61@KRBTEST.COM
I20260926 01:03:58.445191 26220 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.75.60" port: 35451 } attempt: 1
I20260926 01:03:58.445578 26222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:58.445458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.60:35451 (local address 127.0.0.1:58996)
0926 01:03:58.445514 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:58.445540 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.60:35451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 01:03:58.445960 26222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:58.445861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.60:35451 (local address 127.0.0.1:59006)
0926 01:03:58.445905 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:58.445930 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.60:35451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 01:03:58.446287 26220 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.75.60:35451: Network error: Client connection negotiation failed: client connection to 127.3.75.60:35451: connect: Connection refused (error 111)
I20260926 01:03:58.447134 26224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:58.440048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34736 (local address 127.3.75.61:45861)
0926 01:03:58.440122 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:58.440125 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:58.440137 (+ 12us) server_negotiation.cc:407] Connection header received
0926 01:03:58.440189 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:58.440194 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:58.440205 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:58.440266 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:58.440592 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:58.441104 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:58.441818 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:58.442030 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:58.442103 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:58.444251 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:58.444253 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:58.444253 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:03:58.444260 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:58.445887 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:58.446275 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:58.446276 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:58.446276 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:58.446364 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:58.446556 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:58.446557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:58.446557 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:58.446669 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:58.446775 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:58.446975 (+ 200us) server_negotiation.cc:299] Negotiation successful
0926 01:03:58.447050 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 01:03:58.447144 26146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:58.439971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.61:45861 (local address 127.0.0.1:34736)
0926 01:03:58.440027 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:58.440046 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:58.440111 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:58.440279 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:58.440280 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:58.440441 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:03:58.440503 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:58.440506 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:58.441201 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:58.441202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:58.441726 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:58.441727 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:58.442113 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:58.442114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:58.442205 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:58.442736 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:03:58.442741 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:03:58.444139 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:03:58.445964 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:03:58.445965 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:03:58.445966 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:03:58.446187 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:03:58.446418 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:03:58.446419 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:03:58.446420 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:03:58.446475 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:03:58.446768 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:03:58.446769 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:03:58.446819 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 01:03:58.446945 (+ 126us) client_negotiation.cc:240] Negotiation successful
0926 01:03:58.447050 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 01:03:58.447472 26144 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.75.60" port: 35451 } has no permanent_uuid. Determining permanent_uuid...
I20260926 01:03:58.447804 26146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:58.447707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.60:35451 (local address 127.0.0.1:59020)
0926 01:03:58.447755 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:58.447780 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.60:35451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 01:03:58.448168 26146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:58.448078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.60:35451 (local address 127.0.0.1:59028)
0926 01:03:58.448123 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:58.448146 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.60:35451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 01:03:58.448290 26144 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.75.60:35451: Network error: Client connection negotiation failed: client connection to 127.3.75.60:35451: connect: Connection refused (error 111)
I20260926 01:03:58.479466 26144 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.75.60" port: 35451 } attempt: 1
I20260926 01:03:58.479975 26146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:58.479838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.60:35451 (local address 127.0.0.1:59030)
0926 01:03:58.479908 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:58.479948 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.60:35451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 01:03:58.480443 26146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:58.480313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.60:35451 (local address 127.0.0.1:59038)
0926 01:03:58.480369 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:58.480410 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.60:35451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 01:03:58.480577 26144 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.75.60:35451: Network error: Client connection negotiation failed: client connection to 127.3.75.60:35451: connect: Connection refused (error 111)
W20260926 01:03:58.488953 26227 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 01:03:58.489154 26227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:03:58.489172 26227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:03:58.489192 26227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:03:58.490317 26227 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:03:58.490356 26227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:03:58.490370 26227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:03:58.490406 26227 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:03:58.490432 26227 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:03:58.491930 26227 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.75.20:35693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.75.62:36497,127.3.75.61:45861,127.3.75.60:35451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/krb5kdc/kudu_127.3.75.60.keytab
--principal=kudu/127.3.75.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.75.60:35451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.75.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:03:58.492177 26227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:03:58.492408 26227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:03:58.497273 26233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:03:58.497308 26232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:03:58.497367 26227 server_base.cc:1034] running on GCE node
W20260926 01:03:58.497272 26235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:03:58.497689 26227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:03:58.497931 26227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:03:58.499070 26227 hybrid_clock.cc:648] HybridClock initialized: now 1790384638499056 us; error 26 us; skew 500 ppm
Sep 26 01:03:58 dist-test-slave-7v10 krb5kdc[26061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384638, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:03:58.501602 26227 init.cc:377] Logged in from keytab as kudu/127.3.75.60@KRBTEST.COM (short username kudu)
I20260926 01:03:58.502132 26227 webserver.cc:460] Webserver started at http://127.3.75.60:45879/ using document root <none> and password file <none>
I20260926 01:03:58.502460 26227 fs_manager.cc:359] Metadata directory not provided
I20260926 01:03:58.502521 26227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:03:58.502677 26227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:03:58.503789 26227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/minicluster-data/master-2/data/instance:
uuid: "e81fb5434dea4951b0fc3ebfa67cd710"
format_stamp: "Formatted at 2026-09-26 01:03:58 on dist-test-slave-7v10"
server_key: "747fbb6cbb14130bbd3da90d7be88c58"
server_key_iv: "38566fc6905b030ff17f1639f0b1d2fb"
server_key_version: "encryptionkey@0"
I20260926 01:03:58.504170 26227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/minicluster-data/master-2/wal/instance:
uuid: "e81fb5434dea4951b0fc3ebfa67cd710"
format_stamp: "Formatted at 2026-09-26 01:03:58 on dist-test-slave-7v10"
server_key: "747fbb6cbb14130bbd3da90d7be88c58"
server_key_iv: "38566fc6905b030ff17f1639f0b1d2fb"
server_key_version: "encryptionkey@0"
I20260926 01:03:58.505456 26227 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:03:58.506132 26242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:03:58.506300 26227 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:03:58.506394 26227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/minicluster-data/master-2/data,/tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/minicluster-data/master-2/wal
uuid: "e81fb5434dea4951b0fc3ebfa67cd710"
format_stamp: "Formatted at 2026-09-26 01:03:58 on dist-test-slave-7v10"
server_key: "747fbb6cbb14130bbd3da90d7be88c58"
server_key_iv: "38566fc6905b030ff17f1639f0b1d2fb"
server_key_version: "encryptionkey@0"
I20260926 01:03:58.506479 26227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:03:58.514472 26220 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.75.60" port: 35451 } attempt: 2
I20260926 01:03:58.514887 26222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:58.514739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.60:35451 (local address 127.0.0.1:59046)
0926 01:03:58.514800 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:58.514839 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.60:35451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 01:03:58.515254 26222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:58.515164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.60:35451 (local address 127.0.0.1:59056)
0926 01:03:58.515212 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:58.515236 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.75.60:35451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 01:03:58.515372 26220 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.75.60:35451: Network error: Client connection negotiation failed: client connection to 127.3.75.60:35451: connect: Connection refused (error 111)
I20260926 01:03:58.544967 26227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:03:58.547470 26227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:03:58.547636 26227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:03:58.551582 26227 rpc_server.cc:307] RPC server started. Bound to: 127.3.75.60:35451
I20260926 01:03:58.551630 26295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.75.60:35451 every 8 connection(s)
I20260926 01:03:58.551982 26227 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/minicluster-data/master-2/data/info.pb
I20260926 01:03:58.552949 26296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:03:58.554366 26296 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.75.62" port: 36497 } has no permanent_uuid. Determining permanent_uuid...
I20260926 01:03:58.555747 26144 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.75.60" port: 35451 } attempt: 2
Sep 26 01:03:58 dist-test-slave-7v10 krb5kdc[26061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384638, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.62@KRBTEST.COM for kudu/127.3.75.60@KRBTEST.COM
Sep 26 01:03:58 dist-test-slave-7v10 krb5kdc[26061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384638, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.60@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 01:03:58.561344 3372 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktuUsJs/build/release/bin/kudu as pid 26227
I20260926 01:03:58.561447 3372 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktuUsJs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384372073861-3372-0/minicluster-data/master-2/wal/instance
I20260926 01:03:58.561733 3372 external_mini_cluster.cc:1428] Setting key 5e559146913e39219717832751c2a672
I20260926 01:03:58.563972 26147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:58.555029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43266 (local address 127.3.75.62:36497)
0926 01:03:58.555099 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:58.555101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:58.555299 (+ 198us) server_negotiation.cc:407] Connection header received
0926 01:03:58.555749 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:58.555750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:58.555763 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:58.555830 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:58.556627 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:58.557055 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:58.558112 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:58.558301 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:58.558376 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:58.560920 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:58.560921 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:58.560922 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:03:58.560929 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:58.562426 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:58.562919 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:58.562921 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:58.562921 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:58.562989 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:58.563235 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:58.563236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:58.563236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:58.563440 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:58.563519 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:58.563789 (+ 270us) server_negotiation.cc:299] Negotiation successful
0926 01:03:58.563868 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 01:03:58.564028 26146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:58.555970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.60:35451 (local address 127.0.0.1:59064)
0926 01:03:58.556023 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:58.556033 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:58.556071 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:58.556364 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:58.556365 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:58.556490 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:03:58.556525 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:58.556526 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:58.557680 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:58.557681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:58.558125 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:58.558126 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:58.558614 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:58.558615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:58.558674 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:58.559181 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:03:58.559185 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:03:58.560315 (+ 1130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:03:58.562616 (+ 2301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:03:58.562617 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:03:58.562617 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:03:58.562818 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:03:58.563267 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:03:58.563268 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:03:58.563268 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:03:58.563316 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:03:58.563721 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:03:58.563723 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:03:58.563767 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 01:03:58.563838 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 01:03:58.563913 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 01:03:58.564409 26298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:58.555013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.62:36497 (local address 127.0.0.1:43266)
0926 01:03:58.555183 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:58.555218 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:58.555637 (+ 419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:58.555947 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:58.555964 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:58.556312 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:03:58.556536 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:58.556541 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:58.557125 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:58.557126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:58.558019 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:58.558021 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:58.558401 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:58.558402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:58.558526 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:58.559038 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:03:58.559048 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:03:58.560831 (+ 1783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:03:58.562501 (+ 1670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:03:58.562508 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:03:58.562510 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:03:58.562822 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:03:58.563059 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:03:58.563060 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:03:58.563060 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:03:58.563161 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:03:58.563511 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:03:58.563518 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:03:58.563620 (+ 102us) client_negotiation.cc:769] Sending connection context
0926 01:03:58.564064 (+ 444us) client_negotiation.cc:240] Negotiation successful
0926 01:03:58.564197 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":83,"threads_started":1}
I20260926 01:03:58.564410 26299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:58.556044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59064 (local address 127.3.75.60:35451)
0926 01:03:58.556209 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:58.556214 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:58.556230 (+ 16us) server_negotiation.cc:407] Connection header received
0926 01:03:58.556251 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:58.556252 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:58.556274 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:58.556354 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:58.556602 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:58.557600 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:58.558227 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:58.558526 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:58.558603 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:58.560401 (+ 1798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:58.560408 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:58.560409 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:03:58.560415 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:58.562541 (+ 2126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:58.563090 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:58.563093 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:58.563094 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:58.563174 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:58.563384 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:58.563384 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:58.563385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:58.563637 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:58.563728 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:58.563890 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 01:03:58.564012 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":80,"threads_started":1}
I20260926 01:03:58.564947 26296 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.75.61" port: 45861 } has no permanent_uuid. Determining permanent_uuid...
I20260926 01:03:58.566198 26144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2fdca5f39f54b378c7340247e37b35d: Bootstrap starting.
I20260926 01:03:58.566993 26144 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2fdca5f39f54b378c7340247e37b35d: Neither blocks nor log segments found. Creating new log.
Sep 26 01:03:58 dist-test-slave-7v10 krb5kdc[26061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.75.60@KRBTEST.COM
I20260926 01:03:58.567327 26144 log.cc:824] T 00000000000000000000000000000000 P b2fdca5f39f54b378c7340247e37b35d: Log is configured to *not* fsync() on all Append() calls
I20260926 01:03:58.568174 26144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2fdca5f39f54b378c7340247e37b35d: No bootstrap required, opened a new log
I20260926 01:03:58.570278 26144 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2fdca5f39f54b378c7340247e37b35d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2fdca5f39f54b378c7340247e37b35d" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 36497 } } peers { permanent_uuid: "d723d6446940468398b01e4dfa20068c" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 45861 } } peers { permanent_uuid: "e81fb5434dea4951b0fc3ebfa67cd710" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 35451 } }
I20260926 01:03:58.570384 26144 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2fdca5f39f54b378c7340247e37b35d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2fdca5f39f54b378c7340247e37b35d, State: Initialized, Role: FOLLOWER
I20260926 01:03:58.570461 26144 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2fdca5f39f54b378c7340247e37b35d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2fdca5f39f54b378c7340247e37b35d" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 36497 } } peers { permanent_uuid: "d723d6446940468398b01e4dfa20068c" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 45861 } } peers { permanent_uuid: "e81fb5434dea4951b0fc3ebfa67cd710" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 35451 } }
Sep 26 01:03:58 dist-test-slave-7v10 krb5kdc[26061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384638, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.60@KRBTEST.COM for kudu/127.3.75.61@KRBTEST.COM
I20260926 01:03:58.570782 26144 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2fdca5f39f54b378c7340247e37b35d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:03:58.571106 26303 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2fdca5f39f54b378c7340247e37b35d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2fdca5f39f54b378c7340247e37b35d" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 36497 } } peers { permanent_uuid: "d723d6446940468398b01e4dfa20068c" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 45861 } } peers { permanent_uuid: "e81fb5434dea4951b0fc3ebfa67cd710" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 35451 } } }
I20260926 01:03:58.572250 26303 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2fdca5f39f54b378c7340247e37b35d [sys.catalog]: This master's current role is: FOLLOWER
W20260926 01:03:58.573474 26314 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b2fdca5f39f54b378c7340247e37b35d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:03:58.573526 26314 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:03:58.574115 26298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:58.565191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.61:45861 (local address 127.0.0.1:34738)
0926 01:03:58.565229 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:58.565239 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:58.565272 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:58.567104 (+ 1832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:58.567105 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:58.567270 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:03:58.567326 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:58.567328 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:58.568126 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:58.568127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:58.568868 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:58.568871 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:58.569224 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:58.569225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:58.569294 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:58.569622 (+ 328us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:03:58.569627 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:03:58.571017 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:03:58.572902 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:03:58.572903 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:03:58.572904 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:03:58.573104 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:03:58.573462 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:03:58.573463 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:03:58.573463 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:03:58.573530 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:03:58.573820 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:03:58.573821 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:03:58.573866 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 01:03:58.573932 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 01:03:58.574019 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260926 01:03:58.574397 26224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:58.565250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34738 (local address 127.3.75.61:45861)
0926 01:03:58.566962 (+ 1712us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:58.566965 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:58.566974 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:03:58.567002 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:58.567003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:58.567014 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:58.567080 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:58.567431 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:58.568025 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:58.568951 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:58.569150 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:58.569202 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:58.571116 (+ 1914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:58.571117 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:58.571118 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:03:58.571124 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:58.572826 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:58.573315 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:58.573316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:58.573317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:58.573380 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:58.573599 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:58.573600 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:58.573600 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:58.573730 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:58.573792 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:58.574238 (+ 446us) server_negotiation.cc:299] Negotiation successful
0926 01:03:58.574309 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1680}
I20260926 01:03:58.574754 26296 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.75.60" port: 35451 } has no permanent_uuid. Determining permanent_uuid...
I20260926 01:03:58.574942 26301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:58.562249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59070 (local address 127.3.75.60:35451)
0926 01:03:58.562378 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:58.562379 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:58.562387 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:03:58.562408 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:58.562409 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:58.562419 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:58.562490 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:58.563277 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:58.564030 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:58.565111 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:58.565406 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:58.565478 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:58.570731 (+ 5253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:58.570733 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:58.570733 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:03:58.570739 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:58.573372 (+ 2633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:58.573764 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:58.573765 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:58.573766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:58.573833 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:58.574023 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:58.574023 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:58.574024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:58.574140 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:58.574287 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:58.574660 (+ 373us) server_negotiation.cc:299] Negotiation successful
0926 01:03:58.574731 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":111,"thread_start_us":80,"threads_started":1}
I20260926 01:03:58.573886 26074 master_runner.cc:186] Error getting master registration for 127.3.75.61:45861: OK, instance_id { permanent_uuid: "d723d6446940468398b01e4dfa20068c" instance_seqno: 1790384638369563 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 01:03:58.576509 26074 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 26 01:03:58 dist-test-slave-7v10 krb5kdc[26061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384638, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.60@KRBTEST.COM for kudu/127.3.75.60@KRBTEST.COM
Sep 26 01:03:58 dist-test-slave-7v10 krb5kdc[26061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.75.62@KRBTEST.COM
I20260926 01:03:58.584072 26298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:58.576688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.60:35451 (local address 127.0.0.1:59084)
0926 01:03:58.577026 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:58.577035 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:58.577079 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:58.577200 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:58.577200 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:58.577334 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:03:58.577360 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:58.577361 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:58.577981 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:58.577982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:58.578446 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:58.578447 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:58.578838 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:58.578839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:58.578907 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:58.579344 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:03:58.579349 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:03:58.580626 (+ 1277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:03:58.582386 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:03:58.582386 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:03:58.582387 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:03:58.582586 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:03:58.582804 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:03:58.582805 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:03:58.582805 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:03:58.582864 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:03:58.583479 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:03:58.583480 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:03:58.583520 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 01:03:58.583862 (+ 342us) client_negotiation.cc:240] Negotiation successful
0926 01:03:58.583959 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324}
I20260926 01:03:58.584439 26301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:58.576741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59084 (local address 127.3.75.60:35451)
0926 01:03:58.576780 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:58.576782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:58.577127 (+ 345us) server_negotiation.cc:407] Connection header received
0926 01:03:58.577142 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:58.577142 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:58.577149 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:58.577190 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:58.577442 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:58.577906 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:58.578524 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:58.578755 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:58.578838 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:58.580729 (+ 1891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:58.580730 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:58.580731 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:03:58.580737 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:58.582304 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:58.582663 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:58.582664 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:58.582665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:58.582727 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:58.582950 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:58.582951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:58.582952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:58.583386 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:58.583455 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:58.584266 (+ 811us) server_negotiation.cc:299] Negotiation successful
0926 01:03:58.584341 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 01:03:58.585906 26147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:58.576988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43272 (local address 127.3.75.62:36497)
0926 01:03:58.577046 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:58.577048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:58.577056 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:03:58.577075 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:58.577076 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:58.577087 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:58.577152 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:58.577767 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:58.578477 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:58.579488 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:58.579797 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:58.579857 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:58.582767 (+ 2910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:58.582769 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:58.582769 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:03:58.582776 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:58.584497 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:58.584867 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:58.584868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:58.584869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:58.584926 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:58.585123 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:58.585124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:58.585124 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:58.585236 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:58.585370 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:58.585751 (+ 381us) server_negotiation.cc:299] Negotiation successful
0926 01:03:58.585820 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 01:03:58.586050 26296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e81fb5434dea4951b0fc3ebfa67cd710: Bootstrap starting.
I20260926 01:03:58.586745 26296 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e81fb5434dea4951b0fc3ebfa67cd710: Neither blocks nor log segments found. Creating new log.
I20260926 01:03:58.587090 26296 log.cc:824] T 00000000000000000000000000000000 P e81fb5434dea4951b0fc3ebfa67cd710: Log is configured to *not* fsync() on all Append() calls
I20260926 01:03:58.587963 26296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e81fb5434dea4951b0fc3ebfa67cd710: No bootstrap required, opened a new log
I20260926 01:03:58.589644 26296 raft_consensus.cc:348] T 00000000000000000000000000000000 P e81fb5434dea4951b0fc3ebfa67cd710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2fdca5f39f54b378c7340247e37b35d" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 36497 } } peers { permanent_uuid: "d723d6446940468398b01e4dfa20068c" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 45861 } } peers { permanent_uuid: "e81fb5434dea4951b0fc3ebfa67cd710" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 35451 } }
I20260926 01:03:58.589812 26296 raft_consensus.cc:729] T 00000000000000000000000000000000 P e81fb5434dea4951b0fc3ebfa67cd710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e81fb5434dea4951b0fc3ebfa67cd710, State: Initialized, Role: FOLLOWER
I20260926 01:03:58.589905 26296 consensus_queue.cc:260] T 00000000000000000000000000000000 P e81fb5434dea4951b0fc3ebfa67cd710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2fdca5f39f54b378c7340247e37b35d" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 36497 } } peers { permanent_uuid: "d723d6446940468398b01e4dfa20068c" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 45861 } } peers { permanent_uuid: "e81fb5434dea4951b0fc3ebfa67cd710" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 35451 } }
I20260926 01:03:58.590160 26317 sys_catalog.cc:455] T 00000000000000000000000000000000 P e81fb5434dea4951b0fc3ebfa67cd710 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2fdca5f39f54b378c7340247e37b35d" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 36497 } } peers { permanent_uuid: "d723d6446940468398b01e4dfa20068c" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 45861 } } peers { permanent_uuid: "e81fb5434dea4951b0fc3ebfa67cd710" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 35451 } } }
I20260926 01:03:58.590211 26296 sys_catalog.cc:564] T 00000000000000000000000000000000 P e81fb5434dea4951b0fc3ebfa67cd710 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:03:58.591929 26317 sys_catalog.cc:458] T 00000000000000000000000000000000 P e81fb5434dea4951b0fc3ebfa67cd710 [sys.catalog]: This master's current role is: FOLLOWER
Sep 26 01:03:58 dist-test-slave-7v10 krb5kdc[26061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.75.61@KRBTEST.COM
W20260926 01:03:58.592329 26328 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e81fb5434dea4951b0fc3ebfa67cd710: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:03:58.592379 26328 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:03:58.593776 26227 master_runner.cc:186] Error getting master registration for 127.3.75.61:45861: OK, instance_id { permanent_uuid: "d723d6446940468398b01e4dfa20068c" instance_seqno: 1790384638369563 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 01:03:58.596045 26227 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 01:03:58.596438 26224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:58.587363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34742 (local address 127.3.75.61:45861)
0926 01:03:58.587419 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:58.587421 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:58.587429 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:03:58.587449 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:58.587450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:58.587459 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:58.587530 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:58.588110 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:58.588823 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:58.589790 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:58.590097 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:58.590165 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:58.592820 (+ 2655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:58.592822 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:58.592822 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:03:58.592830 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:58.594734 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:58.595198 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:58.595199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:58.595199 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:58.595255 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:58.595526 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:58.595527 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:58.595527 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:58.595650 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:58.596227 (+ 577us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:58.596269 (+ 42us) server_negotiation.cc:299] Negotiation successful
0926 01:03:58.596346 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 01:03:58.597708 3372 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:03:58.598589 26117 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "b2fdca5f39f54b378c7340247e37b35d"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43272
I20260926 01:03:58.598737 26117 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2fdca5f39f54b378c7340247e37b35d [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 01:03:58.598791 26117 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2fdca5f39f54b378c7340247e37b35d [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:03:58.599761 26117 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2fdca5f39f54b378c7340247e37b35d [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2fdca5f39f54b378c7340247e37b35d" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 36497 } } peers { permanent_uuid: "d723d6446940468398b01e4dfa20068c" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 45861 } } peers { permanent_uuid: "e81fb5434dea4951b0fc3ebfa67cd710" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 35451 } }
I20260926 01:03:58.599980 26117 leader_election.cc:288] T 00000000000000000000000000000000 P b2fdca5f39f54b378c7340247e37b35d [CANDIDATE]: Term 1 election: Requested vote from peers d723d6446940468398b01e4dfa20068c (127.3.75.61:45861), e81fb5434dea4951b0fc3ebfa67cd710 (127.3.75.60:35451)
I20260926 01:03:58.600342 26269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b2fdca5f39f54b378c7340247e37b35d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e81fb5434dea4951b0fc3ebfa67cd710"
I20260926 01:03:58.600442 26269 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e81fb5434dea4951b0fc3ebfa67cd710 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:03:58.601652 26269 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e81fb5434dea4951b0fc3ebfa67cd710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2fdca5f39f54b378c7340247e37b35d in term 1.
I20260926 01:03:58.601845 26091 leader_election.cc:302] T 00000000000000000000000000000000 P b2fdca5f39f54b378c7340247e37b35d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2fdca5f39f54b378c7340247e37b35d, e81fb5434dea4951b0fc3ebfa67cd710; no voters:
I20260926 01:03:58.601961 26303 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2fdca5f39f54b378c7340247e37b35d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:03:58.602123 26303 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2fdca5f39f54b378c7340247e37b35d [term 1 LEADER]: Becoming Leader. State: Replica: b2fdca5f39f54b378c7340247e37b35d, State: Running, Role: LEADER
I20260926 01:03:58.602244 26303 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2fdca5f39f54b378c7340247e37b35d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2fdca5f39f54b378c7340247e37b35d" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 36497 } } peers { permanent_uuid: "d723d6446940468398b01e4dfa20068c" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 45861 } } peers { permanent_uuid: "e81fb5434dea4951b0fc3ebfa67cd710" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 35451 } }
I20260926 01:03:58.602720 26303 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2fdca5f39f54b378c7340247e37b35d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2fdca5f39f54b378c7340247e37b35d. Latest consensus state: current_term: 1 leader_uuid: "b2fdca5f39f54b378c7340247e37b35d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2fdca5f39f54b378c7340247e37b35d" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 36497 } } peers { permanent_uuid: "d723d6446940468398b01e4dfa20068c" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 45861 } } peers { permanent_uuid: "e81fb5434dea4951b0fc3ebfa67cd710" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 35451 } } }
I20260926 01:03:58.602807 26303 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2fdca5f39f54b378c7340247e37b35d [sys.catalog]: This master's current role is: LEADER
I20260926 01:03:58.606510 26220 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.75.60" port: 35451 } attempt: 3
I20260926 01:03:58.607445 26339 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:03:58.607654 26339 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:03:58.610522 26301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:58.602974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59092 (local address 127.3.75.60:35451)
0926 01:03:58.603036 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:58.603037 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:58.603047 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:03:58.603068 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:58.603068 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:58.603078 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:58.603135 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:58.603809 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:58.604544 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:58.605541 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:58.605821 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:58.605877 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:58.607403 (+ 1526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:58.607404 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:58.607404 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:03:58.607410 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:58.609137 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:58.609547 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:58.609548 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:58.609548 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:58.609601 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:58.609793 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:58.609793 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:58.609794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:58.609910 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:58.610003 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:58.610359 (+ 356us) server_negotiation.cc:299] Negotiation successful
0926 01:03:58.610428 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
Sep 26 01:03:58 dist-test-slave-7v10 krb5kdc[26061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384638, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.75.61@KRBTEST.COM for kudu/127.3.75.60@KRBTEST.COM
I20260926 01:03:58.613890 26147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:58.602422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43278 (local address 127.3.75.62:36497)
0926 01:03:58.602457 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:58.602459 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:58.602739 (+ 280us) server_negotiation.cc:407] Connection header received
0926 01:03:58.602979 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:58.602981 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:58.602992 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:58.603023 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:58.604166 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:58.604905 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:58.606865 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:58.607200 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:58.607241 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:58.609965 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:58.609967 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:58.609967 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:03:58.609975 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:58.612022 (+ 2047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:58.612668 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:58.612669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:58.612669 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:58.612750 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:58.613110 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:58.613111 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:58.613111 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:58.613265 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:58.613351 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:58.613720 (+ 369us) server_negotiation.cc:299] Negotiation successful
0926 01:03:58.613797 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 01:03:58.614856 26269 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e81fb5434dea4951b0fc3ebfa67cd710 [term 1 FOLLOWER]: Refusing update from remote peer b2fdca5f39f54b378c7340247e37b35d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:03:58.615280 26334 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b2fdca5f39f54b378c7340247e37b35d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e81fb5434dea4951b0fc3ebfa67cd710" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 35451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:03:58.616030 26222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:58.606731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.75.60:35451 (local address 127.0.0.1:59094)
0926 01:03:58.606806 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0926 01:03:58.606816 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:03:58.606850 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:03:58.607085 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:03:58.607085 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:03:58.607254 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:03:58.607603 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:58.607606 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:58.608580 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:58.608581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:58.609458 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:03:58.609461 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:03:58.610042 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:03:58.610043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:03:58.610100 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:58.610531 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:03:58.610537 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:03:58.612100 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:03:58.614272 (+ 2172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:03:58.614273 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:03:58.614274 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:03:58.614494 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:03:58.614701 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:03:58.614702 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:03:58.614702 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:03:58.614763 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:03:58.615264 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:03:58.615265 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:03:58.615342 (+ 77us) client_negotiation.cc:769] Sending connection context
0926 01:03:58.615477 (+ 135us) client_negotiation.cc:240] Negotiation successful
0926 01:03:58.615825 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 01:03:58.616220 26299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:58.606941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59094 (local address 127.3.75.60:35451)
0926 01:03:58.606978 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:58.606979 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:58.606985 (+ 6us) server_negotiation.cc:407] Connection header received
0926 01:03:58.607002 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:58.607003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:58.607011 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:58.607051 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:58.607698 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:58.608484 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:58.609609 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:58.609937 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:58.609993 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:58.612197 (+ 2204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:58.612198 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:58.612199 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:03:58.612205 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:58.614184 (+ 1979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:58.614570 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:58.614571 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:58.614571 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:58.614629 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:58.614834 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:58.614835 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:58.614835 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:58.614988 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:58.615055 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:58.616025 (+ 970us) server_negotiation.cc:299] Negotiation successful
0926 01:03:58.616103 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 01:03:58.616624 26317 sys_catalog.cc:455] T 00000000000000000000000000000000 P e81fb5434dea4951b0fc3ebfa67cd710 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2fdca5f39f54b378c7340247e37b35d. Latest consensus state: current_term: 1 leader_uuid: "b2fdca5f39f54b378c7340247e37b35d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2fdca5f39f54b378c7340247e37b35d" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 36497 } } peers { permanent_uuid: "d723d6446940468398b01e4dfa20068c" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 45861 } } peers { permanent_uuid: "e81fb5434dea4951b0fc3ebfa67cd710" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 35451 } } }
I20260926 01:03:58.616699 26317 sys_catalog.cc:458] T 00000000000000000000000000000000 P e81fb5434dea4951b0fc3ebfa67cd710 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:03:58.617637 26220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d723d6446940468398b01e4dfa20068c: Bootstrap starting.
I20260926 01:03:58.618355 26220 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d723d6446940468398b01e4dfa20068c: Neither blocks nor log segments found. Creating new log.
I20260926 01:03:58.618670 26220 log.cc:824] T 00000000000000000000000000000000 P d723d6446940468398b01e4dfa20068c: Log is configured to *not* fsync() on all Append() calls
I20260926 01:03:58.619493 26220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d723d6446940468398b01e4dfa20068c: No bootstrap required, opened a new log
I20260926 01:03:58.621443 26220 raft_consensus.cc:348] T 00000000000000000000000000000000 P d723d6446940468398b01e4dfa20068c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2fdca5f39f54b378c7340247e37b35d" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 36497 } } peers { permanent_uuid: "d723d6446940468398b01e4dfa20068c" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 45861 } } peers { permanent_uuid: "e81fb5434dea4951b0fc3ebfa67cd710" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 35451 } }
I20260926 01:03:58.621567 26220 raft_consensus.cc:729] T 00000000000000000000000000000000 P d723d6446940468398b01e4dfa20068c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d723d6446940468398b01e4dfa20068c, State: Initialized, Role: FOLLOWER
I20260926 01:03:58.621639 26220 consensus_queue.cc:260] T 00000000000000000000000000000000 P d723d6446940468398b01e4dfa20068c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2fdca5f39f54b378c7340247e37b35d" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 36497 } } peers { permanent_uuid: "d723d6446940468398b01e4dfa20068c" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 45861 } } peers { permanent_uuid: "e81fb5434dea4951b0fc3ebfa67cd710" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 35451 } }
I20260926 01:03:58.621882 26224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:58.602727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34754 (local address 127.3.75.61:45861)
0926 01:03:58.602763 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:58.602764 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:58.610835 (+ 8071us) server_negotiation.cc:407] Connection header received
0926 01:03:58.610983 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:58.610984 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:58.610994 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:58.611023 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:58.611563 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:58.612257 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:58.613251 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:58.613519 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:58.613557 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:58.618997 (+ 5440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:03:58.618998 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:03:58.618998 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:03:58.619005 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:03:58.620666 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:58.621063 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:58.621064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:58.621065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:58.621131 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:03:58.621322 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:03:58.621322 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:03:58.621323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:03:58.621446 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:03:58.621529 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:58.621709 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 01:03:58.621780 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 01:03:58.621928 26220 sys_catalog.cc:564] T 00000000000000000000000000000000 P d723d6446940468398b01e4dfa20068c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:03:58.622191 26193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b2fdca5f39f54b378c7340247e37b35d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d723d6446940468398b01e4dfa20068c"
I20260926 01:03:58.622299 26193 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d723d6446940468398b01e4dfa20068c [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:03:58.623152 26193 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d723d6446940468398b01e4dfa20068c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2fdca5f39f54b378c7340247e37b35d in term 1.
I20260926 01:03:58.624589 26303 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2fdca5f39f54b378c7340247e37b35d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b2fdca5f39f54b378c7340247e37b35d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2fdca5f39f54b378c7340247e37b35d" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 36497 } } peers { permanent_uuid: "d723d6446940468398b01e4dfa20068c" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 45861 } } peers { permanent_uuid: "e81fb5434dea4951b0fc3ebfa67cd710" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 35451 } } }
I20260926 01:03:58.624707 26303 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2fdca5f39f54b378c7340247e37b35d [sys.catalog]: This master's current role is: LEADER
I20260926 01:03:58.624782 26340 mvcc.cc:205] Tried to move back new op lower bound from 7333415479764156416 to 7333415479715590144. Current Snapshot: MvccSnapshot[applied={T|T < 7333415479764156416}]
I20260926 01:03:58.623513 26192 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d723d6446940468398b01e4dfa20068c [term 1 FOLLOWER]: Refusing update from remote peer b2fdca5f39f54b378c7340247e37b35d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:03:58.624799 26317 sys_catalog.cc:455] T 00000000000000000000000000000000 P e81fb5434dea4951b0fc3ebfa67cd710 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b2fdca5f39f54b378c7340247e37b35d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2fdca5f39f54b378c7340247e37b35d" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 36497 } } peers { permanent_uuid: "d723d6446940468398b01e4dfa20068c" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 45861 } } peers { permanent_uuid: "e81fb5434dea4951b0fc3ebfa67cd710" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 35451 } } }
I20260926 01:03:58.624890 26317 sys_catalog.cc:458] T 00000000000000000000000000000000 P e81fb5434dea4951b0fc3ebfa67cd710 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:03:58.625031 26334 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b2fdca5f39f54b378c7340247e37b35d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d723d6446940468398b01e4dfa20068c" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 45861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:03:58.625193 26341 mvcc.cc:205] Tried to move back new op lower bound from 7333415479764156416 to 7333415479715590144. Current Snapshot: MvccSnapshot[applied={T|T < 7333415479764156416}]
I20260926 01:03:58.625499 26344 sys_catalog.cc:455] T 00000000000000000000000000000000 P d723d6446940468398b01e4dfa20068c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2fdca5f39f54b378c7340247e37b35d" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 36497 } } peers { permanent_uuid: "d723d6446940468398b01e4dfa20068c" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 45861 } } peers { permanent_uuid: "e81fb5434dea4951b0fc3ebfa67cd710" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 35451 } } }
I20260926 01:03:58.625716 26344 sys_catalog.cc:458] T 00000000000000000000000000000000 P d723d6446940468398b01e4dfa20068c [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:03:58.627530 26355 catalog_manager.cc:1259] Loaded cluster ID: 80cf1e3167fd45859c3909ee55a3ba67
I20260926 01:03:58.627586 26355 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d723d6446940468398b01e4dfa20068c: loading cluster ID for follower catalog manager: success
W20260926 01:03:58.627704 26355 catalog_manager.cc:1576] T 00000000000000000000000000000000 P d723d6446940468398b01e4dfa20068c: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260926 01:03:58.627748 26355 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:03:58.627853 26344 sys_catalog.cc:455] T 00000000000000000000000000000000 P d723d6446940468398b01e4dfa20068c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2fdca5f39f54b378c7340247e37b35d. Latest consensus state: current_term: 1 leader_uuid: "b2fdca5f39f54b378c7340247e37b35d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2fdca5f39f54b378c7340247e37b35d" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 36497 } } peers { permanent_uuid: "d723d6446940468398b01e4dfa20068c" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 45861 } } peers { permanent_uuid: "e81fb5434dea4951b0fc3ebfa67cd710" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 35451 } } }
I20260926 01:03:58.627899 26358 sys_catalog.cc:455] T 00000000000000000000000000000000 P d723d6446940468398b01e4dfa20068c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b2fdca5f39f54b378c7340247e37b35d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2fdca5f39f54b378c7340247e37b35d" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 36497 } } peers { permanent_uuid: "d723d6446940468398b01e4dfa20068c" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 45861 } } peers { permanent_uuid: "e81fb5434dea4951b0fc3ebfa67cd710" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 35451 } } }
I20260926 01:03:58.627986 26358 sys_catalog.cc:458] T 00000000000000000000000000000000 P d723d6446940468398b01e4dfa20068c [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:03:58.627918 26344 sys_catalog.cc:458] T 00000000000000000000000000000000 P d723d6446940468398b01e4dfa20068c [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:03:58.635362 26339 catalog_manager.cc:1347] Generated new cluster ID: 80cf1e3167fd45859c3909ee55a3ba67
I20260926 01:03:58.635439 26339 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:03:58.655740 26334 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2fdca5f39f54b378c7340247e37b35d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b2fdca5f39f54b378c7340247e37b35d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2fdca5f39f54b378c7340247e37b35d" member_type: VOTER last_known_addr { host: "127.3.75.62" port: 36497 } } peers { permanent_uuid: "d723d6446940468398b01e4dfa20068c" member_type: VOTER last_known_addr { host: "127.3.75.61" port: 45861 } } peers { permanent_uuid: "e81fb5434dea4951b0fc3ebfa67cd710" member_type: VOTER last_known_addr { host: "127.3.75.60" port: 35451 } } }
I20260926 01:03:58.655884 26334 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2fdca5f39f54b378c7340247e37b35d [sys.catalog]: This master's current role is: LEADER
I20260926 01:03:58.658244 26339 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:03:58.659060 26339 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:03:58.667305 26339 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2fdca5f39f54b378c7340247e37b35d: Generated new TSK 0
I20260926 01:03:58.667501 26339 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 01:03:58.684927 26147 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:03:58.683409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43282 (local address 127.3.75.62:36497)
0926 01:03:58.683450 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:58.683452 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:58.684285 (+ 833us) server_negotiation.cc:407] Connection header received
0926 01:03:58.684387 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:58.684388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:58.684400 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:58.684430 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:58.684860 (+ 430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43282: BlockingRecv error: recv got EOF from 127.0.0.1:43282 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 01:03:58.685847 26224 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:03:58.683745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34758 (local address 127.3.75.61:45861)
0926 01:03:58.683782 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:58.683784 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:58.685138 (+ 1354us) server_negotiation.cc:407] Connection header received
0926 01:03:58.685410 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:58.685411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:58.685421 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:58.685450 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:58.685788 (+ 338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34758: BlockingRecv error: recv got EOF from 127.0.0.1:34758 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260926 01:03:58.687295 26299 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:03:58.684056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59096 (local address 127.3.75.60:35451)
0926 01:03:58.684116 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:58.684119 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:58.685266 (+ 1147us) server_negotiation.cc:407] Connection header received
0926 01:03:58.685334 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:58.685336 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:58.685347 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:58.685416 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:58.687239 (+ 1823us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59096: BlockingRecv error: recv got EOF from 127.0.0.1:59096 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260926 01:03:58.689657 26224 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:03:58.688852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34760 (local address 127.3.75.61:45861)
0926 01:03:58.688887 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:58.688889 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:58.689134 (+ 245us) server_negotiation.cc:407] Connection header received
0926 01:03:58.689246 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:58.689247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:58.689264 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:58.689295 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:58.689611 (+ 316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34760: BlockingRecv error: recv got EOF from 127.0.0.1:34760 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260926 01:03:58.692587 26224 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:03:58.691802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34772 (local address 127.3.75.61:45861)
0926 01:03:58.691834 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:58.691835 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:58.692095 (+ 260us) server_negotiation.cc:407] Connection header received
0926 01:03:58.692195 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:58.692196 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:58.692206 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:58.692235 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:58.692541 (+ 306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34772: BlockingRecv error: recv got EOF from 127.0.0.1:34772 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260926 01:03:58.696611 26224 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:03:58.695738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34784 (local address 127.3.75.61:45861)
0926 01:03:58.695774 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:58.695776 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:58.696053 (+ 277us) server_negotiation.cc:407] Connection header received
0926 01:03:58.696187 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:58.696188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:58.696199 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:58.696230 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:58.696561 (+ 331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34784: BlockingRecv error: recv got EOF from 127.0.0.1:34784 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260926 01:03:58.702665 26224 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:03:58.701767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34792 (local address 127.3.75.61:45861)
0926 01:03:58.701807 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:58.701809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:58.702097 (+ 288us) server_negotiation.cc:407] Connection header received
0926 01:03:58.702204 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:58.702205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:58.702216 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:58.702247 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:58.702611 (+ 364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34792: BlockingRecv error: recv got EOF from 127.0.0.1:34792 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260926 01:03:58.712944 26224 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:03:58.711896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34798 (local address 127.3.75.61:45861)
0926 01:03:58.711944 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:58.711946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:58.712288 (+ 342us) server_negotiation.cc:407] Connection header received
0926 01:03:58.712412 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:58.712413 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:58.712427 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:58.712464 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:58.712880 (+ 416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34798: BlockingRecv error: recv got EOF from 127.0.0.1:34798 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 01:03:58.731379 26224 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:03:58.730295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34812 (local address 127.3.75.61:45861)
0926 01:03:58.730341 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:58.730343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:58.730659 (+ 316us) server_negotiation.cc:407] Connection header received
0926 01:03:58.730780 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:58.730782 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:58.730796 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:58.730831 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:58.731319 (+ 488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34812: BlockingRecv error: recv got EOF from 127.0.0.1:34812 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 01:03:58.765453 26224 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:03:58.764661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34826 (local address 127.3.75.61:45861)
0926 01:03:58.764711 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:58.764713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:58.764867 (+ 154us) server_negotiation.cc:407] Connection header received
0926 01:03:58.764949 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:58.764950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:58.764967 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:58.765028 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:58.765384 (+ 356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34826: BlockingRecv error: recv got EOF from 127.0.0.1:34826 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 01:03:58.831332 26224 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:03:58.830509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34840 (local address 127.3.75.61:45861)
0926 01:03:58.830603 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:58.830605 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:58.830685 (+ 80us) server_negotiation.cc:407] Connection header received
0926 01:03:58.830734 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:58.830735 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:58.830745 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:58.830811 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:58.831253 (+ 442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34840: BlockingRecv error: recv got EOF from 127.0.0.1:34840 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260926 01:03:58.961239 26224 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:03:58.960546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34856 (local address 127.3.75.61:45861)
0926 01:03:58.960601 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:58.960603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:58.960692 (+ 89us) server_negotiation.cc:407] Connection header received
0926 01:03:58.960755 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:58.960756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:58.960766 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:58.960809 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:58.961140 (+ 331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34856: BlockingRecv error: recv got EOF from 127.0.0.1:34856 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260926 01:03:59.219272 26224 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:03:59.218604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34864 (local address 127.3.75.61:45861)
0926 01:03:59.218685 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:59.218687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:59.218709 (+ 22us) server_negotiation.cc:407] Connection header received
0926 01:03:59.218787 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:59.218788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:59.218798 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:59.218839 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:03:59.219200 (+ 361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34864: BlockingRecv error: recv got EOF from 127.0.0.1:34864 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 01:03:59.592792 26328 catalog_manager.cc:1259] Loaded cluster ID: 80cf1e3167fd45859c3909ee55a3ba67
I20260926 01:03:59.592864 26328 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e81fb5434dea4951b0fc3ebfa67cd710: loading cluster ID for follower catalog manager: success
I20260926 01:03:59.593623 26328 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e81fb5434dea4951b0fc3ebfa67cd710: acquiring CA information for follower catalog manager: success
I20260926 01:03:59.593873 26328 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e81fb5434dea4951b0fc3ebfa67cd710: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 01:03:59.628839 26355 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d723d6446940468398b01e4dfa20068c: acquiring CA information for follower catalog manager: success
I20260926 01:03:59.629139 26355 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d723d6446940468398b01e4dfa20068c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 01:03:59.734827 26427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:59.732395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34866 (local address 127.3.75.61:45861)
0926 01:03:59.732530 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:59.732532 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:59.732542 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:03:59.732604 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:59.732604 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:59.732653 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:59.732736 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:03:59.732875 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:59.733366 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:59.734081 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:59.734274 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:59.734325 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:59.734473 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:03:59.734577 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:03:59.734634 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:59.734691 (+ 57us) server_negotiation.cc:299] Negotiation successful
0926 01:03:59.734707 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":68,"threads_started":1}
I20260926 01:03:59.738879 26434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:59.736656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59112 (local address 127.3.75.60:35451)
0926 01:03:59.736785 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:59.736787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:59.736795 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:03:59.736829 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:59.736830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:59.736866 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:59.736935 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:03:59.737032 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:59.737524 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:59.738190 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:59.738376 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:59.738426 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:59.738548 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:03:59.738649 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:03:59.738706 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:59.738737 (+ 31us) server_negotiation.cc:299] Negotiation successful
0926 01:03:59.738755 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":68,"threads_started":1}
I20260926 01:03:59.743541 26434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:59.740727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59122 (local address 127.3.75.60:35451)
0926 01:03:59.740765 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:59.740767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:59.740774 (+ 7us) server_negotiation.cc:407] Connection header received
0926 01:03:59.740797 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:59.740798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:59.740810 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:59.740868 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:03:59.740991 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:59.741593 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:59.742524 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:59.742823 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:59.742930 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:59.743132 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:03:59.743163 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:03:59.743216 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:59.743442 (+ 226us) server_negotiation.cc:299] Negotiation successful
0926 01:03:59.743466 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 01:03:59.743773 26427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:59.740779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34868 (local address 127.3.75.61:45861)
0926 01:03:59.740814 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:59.740817 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:59.741030 (+ 213us) server_negotiation.cc:407] Connection header received
0926 01:03:59.741127 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:59.741128 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:59.741138 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:59.741166 (+ 28us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:03:59.741345 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:59.742036 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:59.742998 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:59.743286 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:59.743342 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:59.743504 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:03:59.743530 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:03:59.743570 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:59.743690 (+ 120us) server_negotiation.cc:299] Negotiation successful
0926 01:03:59.743713 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 01:03:59.744151 26441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:03:59.740605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43290 (local address 127.3.75.62:36497)
0926 01:03:59.740720 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 01:03:59.740722 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:03:59.740813 (+ 91us) server_negotiation.cc:407] Connection header received
0926 01:03:59.740913 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:03:59.740914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:03:59.740968 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:03:59.741050 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:03:59.741149 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:59.741887 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:59.742817 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:03:59.743229 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:03:59.743291 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:03:59.743708 (+ 417us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:03:59.743818 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:03:59.743879 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 01:03:59.744017 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 01:03:59.744042 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":54,"threads_started":1}
I20260926 01:03:59.750448 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 26074
I20260926 01:03:59.755842 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 26151
I20260926 01:03:59.760535 3372 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktuUsJs/build/release/bin/kudu with pid 26227
2026-09-26T01:03:59Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2900 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5747 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (267694 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST